Update: 7/26/12 Hootsuite has changed ow.li from a 302 redirect to a 301 redirect. Thanks, Hootsuite, for reading! (Link to comment from Matt)
Preface: How Normal URL Shorteners Work
Hang tight, folks! Before we get into the whooo-done-it part of this post, here's a basic illustration of how URL shorteners work.
It's pretty simple actually, on the surface.
- You need a short domain name (like bit.ly - which is possible because of the .ly extension)
- A random URL (or custom "vanity" if you choose) is created on the short domain like bit.ly/h1oiSn/ or bity.ly/example-url/ (both fake)
- A visit to that URL sends the user via 301 redirect or 302 redirect to the page it's linked to.
Here's a real example of a URL I recently shortened with Bitly:
.png)
See? NORMAL. And Bit.ly uses a 301 redirect which is also most liked, since 301's pass link equity.
And the Bitly URL should (ideally) never rank above the final "real" URL.
Danny Sullivan's post, although a bit outdated now, shows (of the shorteners from that time) which services use 301s and which use 302s. Note that he highly advises against the use of URLs with 302s. Remember that for later :-)
If you want to examine more details before we begin, you can check out the wikipedia page on URL shortening. However, the point of this post is to analyze a particular SERP which returns a shortened URL instead of the normal URL.
Which One Of These Things Doesn't Belong?
For one, startupcity.org should probably rank #1 - implied brand search. But... that's not quite it. See it?
Perhaps if I restate it.
Here's a Hint - Whoooo Doesn't Belong?
If You Said "The Owl" or "Owl.li" - Good Job!
No prize though I'm afraid.
Strange result credit: tweet by Rand.
There are two things wrong with this screenshot.
- An owl.li URL is ranking in the first page on the SERPs
- Startupcity.org is not ranking as well as you'd think, as this is definitely an implied brand search.
Needless to say, Rand's tweet highlighting this strange behavior certainly led to the idea for this post.
Initial Hypothesis: Owl.li URLs are NOT normal
But what's different about them? What's causing one to rank on the first page?
Who's at fault - Google, owl.li, or the site owner?
Let's see if we can find out!! In the process, we'll examine the implications of using owl.li and come up with some best practices that you can follow to avoid this happening on your site.
The first thing I'd like to invite you to do is to join me on a guided video tour of a bunch of screenshots I took while investigating this.
The Evidence In a Screencast
Oh boy, this is a multimedia extravaganza on the Moz blog! I decided to walk through all my bits of evidence in a screencast - to give you a quick overview of the investigative process behind this.
Video 1/2 - you'll want to expand to full screen
Video 2/2 - again, full screen will look best!
Well now... those videos were fun, yes? Alright, let's examine the biggest takeaways from this first look.
Top 5 Highlights From the Video
You didn't want to watch the videos? (Sadface) OK. Here are the 5 most significant screencaps:
1. What's the Redirect Path To the Final URL?
This naturally was my first question. How is owl.li executing the redirects? And what a fabulous opportunity to use this new redirect checker chrome plugin by Ayima. The redirect path of this owl.li URL is as follows:
It's a 302 redirect followed by three 301 redirects. I know redirect chains are not the best thing in the world, so could this be causing the strange indexation? I file that question in my brain for later.
2. Does StartupCity Have Internal Links Sending Mixed Signals?
You should ALWAYS internally link to the current version of your URLs. This makes your site faster, it prevents things from breaking in the future, and allows only external links/bookmarks to be passed through redirects.
Sometimes internal links pointing to pages which then 301 can send mixed signals. Sure enough, there's some of that happening on startupcity.org
However, I'm not entirely convinced this is causing Google to actually rank the owl.li page.
3. What Does Google's Documentation Say About Redirects?
Google states pretty clearly that they can and do index 302's.
I do find it interesting that Google doesn't say either way if a 301 can be indexed.
4. Can I Get BOTH URLs To Return in a Search?
I switched around the keywords, and now we have both URLs ranking on the first page. Yes, the owl.li URL AND the startupcity.org URL are both on the first page.
5. How Do You Make an Owl.li URL?
You'll see in the video that I had a few unsuccessful attempts at making an owl.li URL (had no idea, never done it before!). I finally figured out where owl.li links come from. I'm sure some of you already know, but as I am not a heavy HootSuite user, I had to look. (Note: this image is not in the video. So consider this a bonus? Unless you're watching the video - then you're missing out.)
- First put the link in the link field
- Before shortening click the gear (advanced)
- Select owl.li from the drop down
Who to Blame? Google, Owl.li, or the Site Owner?
The following conclusions and recommendations are my personal thoughts - I would follow them myself and refer them to clients. However, I have to be completely honest. I'm not 100% sure why Google is ranking that owl.li result for [startupcity 100].
I do have some strong hypotheses and tips for site owners.
Owl.li Was Created as an Alternative to the .ly Domain
I finally found some record of when and why the owl.li shortener was put into place.
- Here's the post directly from HootSuite - to sum up, they state to have initiated it because of concerns over the .ly top level domain and civil unrest in Libya (the country from which the TLD belongs to). Users of HootSuite were given the option of choosing their shortener in April of 2011
- Here's an article from the Social Times - the article more or less sums up the move to owl.li for safety and all that.
Now I know WHY this was put into place, but still don't know why they chose to use 302's.
Also, they claim users can avoid the .ly TLD - this isn't entirely true though, because everything still passes through ow.ly as a redirect.
Owl.li Sends You Through a 302 and Then 301 Redirect
Owl.li is not like other short URL services. It sends you through a 302 and then a 301.
This is perhaps the biggest takeaway of all, which is worth restating. Owl.li sends their links via 302, through ow.ly which then 301 to the final page. This, again, seems like a double whammy. Not only is the "double redirect" not necessary, the 302 means none of the link equity is being passed on to your page.
Again, I would like to know - why does HootSuite tell us owl.li is an alternative to ow.ly (to avoid the .ly) but then direct all URLs through ow.ly anyway?
Google's Side - Possible Reasons/Implications of Shortened URLs In the Index
Redirects alone don't prevent indexation - Google does not say they don't index 301's or 302's. They state in the affirmative that they may index 302's - and they don't state anything either way about 301's. So it does make sense as to why the shortened URLs are in the index. (Please note that Bing does not index 302 redirects, according to this article from 2010 - which is likely why there are almost no owl.li URLs indexed).
Ranking signal - They were likely indexed because they were tweeted. Perhaps further indication of Google using Twitter as a ranking signal? Index the short URLs and assign a page value to them (by number of tweets, inbound links, etc.)? There's really no value for the user to have them in the index, right?
Destination pages had issues - The pages they pointed to had some internal issues - like the IP addresses not resolving, or the internal linking, or large chains of redirects.
A short URL ranking well Is an anomaly - However, the startupcity.org result was the ONLY one that actually ranked above its destination page. All the other ones may be indexed, but I have never seen a short URL ranking on page one before. (If you have I'd be interested - tweet me @dan_shure with a screencap).
Site Owners - Follow These Best Practices
Ultimately, I don't blame the site owner in this case (or in many others). But there are some best practices to follow that can reduce the risk of these issues (or others) appearing in the SERPs.
1. Reduce and ideally eliminate chains of redirects
- Use the new tool by Ayima to check for redirect chains on a page by page basis.
- To check redirect headers a bit more intensely, use URI Valet (set to Googlebot for indexation questions).
- You can even use Screaming Frog SEO Spider to check for redirects.
To eliminate redirect chains; instead of Page 1->Page 2->Page 3
Use
Page 1->Page 3
Page 2->Page 3
2. Use noindex, the URL removal request, or robots.txt to permanently get a page deindexed
Google will index 301 and 302 redirected URLs. If the page still exists, use a meta noindex tag on the page and request removal with webmaster tools. If the page no longer exists, do the URL removal request and use robots.txt to keep it from getting back in the index.
3. Keep your site's internal links updated.
As mentioned, this helps to;
- Make your site faster
- Eliminate mixed signals
- Prevent things from getting broken in the future
- Only external links/bookmarks have to get passed through redirects
Point your users at the current version of your URLs whenever possible. Anything from pointing to the homepage as /index.php when its just / - always keep your internal links updated.
Use Screaming Frog to crawl the site and look for any that need fixin'
4. Use Ow.ly in preference to Owl.li
We have seen that by using owl.li you're STILL using Ow.ly - because the ow.li 302 redirects through ow.ly. Who knows if this will always be the case, but why send your links through an extra redirect and also risk the owl.li URL showing in the SERP of the destination page?
So Whooo's Responsible?
Right - I never did say :-)
It sure would be most fun to point blame at one particular party - but in this case it's completely a team effort.
- Google - is indexing thousands of owl.li URLs
- Owl.li - is sending all those URLs through a 302 and then a 301
- Startupcity.org - has additional 301 redirects leading to that page, in addition to internal links not pointing at the most current page.
I believe it is the combination of these three factors that's causing this, but my findings are certainly not scientifically conclusive. I always encourage you to do your own questioning and investigating.
So site owners, follow the best practices above, and remember how interesting it is to bring this third service of URL shortening into the mix to contribute to these sorts of issues.
Further Reading On URL Shortening
It seems this topic was of much larger discussion around 2009. Understandably so, since its kind of "old news" now. But is it really old news? Danny Sullivan says that it's really time for an updated look into this topic.
- As cited above, Danny Sullivan did the most in depth piece I could find on URL shorteners, but back in 2009
- Matt Cutts did a quick Webmaster Tools video on this
I personally hope this post may get some other people to look more into this topic, starting with the comments. I'd love to hear your thoughts on this topic.
And...
I'll be at MozCon... come say hi!
or say hi out in the interwebs :)
Update - Was About to Hit Publish, and then...
You thought this post was done? Well, so did I! And then I did one last search for startupcity.org - only to find the whole site now redirects to another URL, startupseattle.com
So if you go and try to research this yourself - it seems like this case is changing by the hour! Definitely an interesting one to follow.
Good detective work. But while reading your post a question arises in my mind. Why you discard social signals? I think Social influence work as well. New domains come up in SERP and after 1 hr they move back sometime within in seconds. If group of twitter influencer start tweeting it, it probably stay there for a long time.
I think this URL should be buried in SERP but due to excessive tweeting cause this problem.
My scenario is only an assumption may be I am completely wrong or may be partially.
Hi Asif
I think that is a good point - and in fact one thing I noticed was the ranking of shortened URLs in SERPs (doing a site:owl.li search for example) change constantly. I nearly included the factor of time in the post but decided I had to draw the information line somewhere :)
But definitely, to your point, I do think who tweets something may impact the ranking of an owl.li URL - not only that but how many, how often, velocity, etc.
Excellent point, and one I'd be interested in exploring!
-Dan
Also... the second video examines tweets a little bit ;-)
Good post Dan! I just finished cleaning up some Sitelinks for a site which included a URL (a misspelling) that was non-existent. A 404 Session Timeout page was showing up in Sitelinks because all 404s were apparently being 302 redirected to the timeout page.
This is a practice that needs to change - you can 301 redirect 404s to a "Page Not Found" if you like, but don't let crawlers think they should be indexing non-existent content!
Only 302 if you know a page is going to be "missing" for a while, but will return; it's either temporary or it's not :).
Ok now we're let in on what your "million dollar idea" was the other day... nice work man! You're a smart guy, it's nice to see you putting stuff out there.
Using a url shortener is quite risky, because you never know when they'll change the links or when they'll go belly up. Now we see another risk presented by shorteners.
Here's why I use URL shorteners (specifically owl.ly)
1. The shelf life of these links are short anyway - I use them in a tweet or a Google+ post and they're forgotten about in hours, maybe days if I'm really lucky.
2. They enable you to add more content to Twitter posts - This could help get more clicks and retweets.
3. They take away any bias towards the domain name - hey you might have some good content but a horrible domain name. Maybe it looks spammy or cheesy but actually has good content on it. With a URL shortener you don't have someone's bias, a link is a link.
4. It's almost required by Hootsuite - I use hootsuite to automate and organize my posts. I'm not a spammer, I just find so much stuff that if I share it all during the day people will get annoyed. Plus believe it or not I work during the day and have a life, so I schedule tweets. When you do that with hootsuite it requires many links to be shortened and they do it automatically.
So that's why I use them but it's only for social media purposes, and most of the time I'm not linking to my own pages anyway. I agree that you should never use them on your site for internal linking, or external linking on any other site.
This post brings up some really interesting stuff to think about and I think there's a problem here we need to start looking at.
great post, love the step by step explanation, +1, tweet, Like for this :)
+1, I agree with Khalid. Great post Dan! It's some of these minor SEO aspects that people don't really think about. I've never thought about this but it makes total sense. Great work putting together something outside the normal realm of SEO blog posts.
Agree great post and explanation the the URL shorten services!
Hi Dan,
I definitely echo all of the comments stating what a great post this is. Such an amazingly thorough analysis.
We certainly want to ensure we are contributing, not hindering, a better experience for the searcher and feedback like this keeps us all honest and in check. We have made it so that Owl.li does a 301 redirect now and will be interested to see the impact this makes on the SERPs. Let us know your findings as well.
Thanks again Dan!
Wow - thanks so much for reading the post and changing things around. I'd be extremely curious to know of any differences it makes, so will definitely check in again!
Hey Den, you concluded lots of things in this video and i really appreciate this but we didn't conclude anything any result something else??? is that beneficial or not???? will this make a negative impact???
Do you mean will the owl.li url in the SERP cause anything negative?
I think is being in the SERP is a symptom of the real underlying issue, which is that owl.li redirect via 302 to ow.ly which then 301s to the page.
And if can rank in the SERP above the real page, then something is giving that owl.li URL more weight - which isn't right. The value should be passed on to the final page by using a 301 instead. Why they chose to use 302s isn't clear, but the 302 doesn't pass link value.
Why do people use link shorteners anymore? Most social platforms, for example, do automatic link shortening. Certainly they are now relatively useless.
1. Easy to remember
2. To email to a friend without a long URL
3. To be cool (or try, anyway)
Yes, but most of these social services use a third party service to shorten them.
Some link shorteners give you built in analytics as well. Instant CT information.
2 different things there mate . Granted Social Platforms to their own ( or 3rd party URL shortening services ) but that does not render the service itself useless . It's automated and users don't have to take the extra step ( which is great ) , but the end result is people still use them as its useful.
Wow, great post Dan! I really enjoyed the detail, images, and screencast. I learned a lot about how the link shortners work from this post. Also, I got the ayima Chrome plugin, which looks pretty sweet.
Yes - I love the plugin.
Just to note, the Moz toolbar plugin does that as well. Its under page analysis->page attributes. I like the Ayima one because it highlights automatically when a redirect is detected.
Great detective work Dan.
"Screencast Wednesdays", has a nice ring to it.
You shouldn't put cute Image like that...couldn't concentrate on Post.I think the first cat from Right have suspected about owl (Look at his face)
Post was great but I will be giving the thumb up for the Image.
Great analysis of a tricky redirect issue, but why would you recommend people use ow.ly over ow.li. In the example you gave, the site had 2 listings on the first page of Google thanks to ow.li-- that sounds good to me!
Because owl.li works via a 302 redirect which means owl.li retains all link equity. Also, when I eliminated owl.li from the search (in the video) the correct page ranked where it should have, which means owl.li is not causing the ranking, its replacing the ranking of the correct URL.
Great post. Although I haven't seen this with any of my clients, I do have some that use HootSuite for tweets. At best, they are probably diluting their link authority somewhat. At worst, if they are successful in creating a buzz through twitter, they don't reap the full benefit because owl.li gets the credit, not the original site.
Great actionable blog. Thanks.
A great post and it's really got me thinking about shortners. Is it safer to use just your own link or a shortner when using a third party application like hootsuite or tweetdeck? If anyone has some thoughts on this, I would love to know.
Hi Dan, Gr8 post. Thanks for bringing it to my attention.
This is why I try never to use shortners and redirects - they go wrong when there are too many in the chain - I think Matt Cutts did a video on this.
I use https://www.ttwi.st which is a premium URL shortener made for Internet Marketers. They are Clickbank friendly, have great tracking and have their own API. Your stats are also your own rather than having links shared with other people like some other URL shorteners. No threats of the .ly or 302's.
This really is a great post.
Long tail and short tail domain name have different impact on SEO and both are treated differently in search engines. It's recommended to choose the short tail domain name as well as a short tail keyword to get good ranking in Google. While your research regarding the domain name is incredably appreciatable.
Thanks for sharing :)
I guess its a 302 a temp so there would be no reason not to be indexed, if its a 301 it most certainly should not be indexed as permanent.
Screenshots ideas was best part of this post thanks for this all information
So, I skimmed most of the comments and didn't see an answer to my question,so I hope I didn't miss something... but here goes:
Does open site explorer recognize social shares that use URL shorteners instead of the direct URL?
Of course there is value in using Hootsuite (shorter posts = longer Tweets, scheduling, etc) but if by using Hootsuite which forces me to use their URL shortener I am #1) getting diminished link credit from the 301/302 redirecting and #2) not getting social share counts to show to my client... then I might just have to stop using Hootsuite and go direct and input the whole URL.
Side note: It's really unfortunate that Hootsuite has not adapted to the Twitter auto long URL shortner (not a separate URL like bit.ly or owl.ly but like URL.com/xyz... ) as that would solve this problem altogether.
Hey Jenny
I'm not entirely sure how OSE factors that into their metrics. If someone links to you with bitly, no matter through social or just a regular 'ol hyperlink, you won't see it in your inbound links, because it's a cross domain 301 redirect.
However, you can see everyone linking to bitly links here: https://www.opensiteexplorer.org/links?site=bitly.com%2F - so OSE sorta counts those as links for bitly.
BUT the important thing to keep in mind, is that these are 301 redirects - and so in reality you get the value of these links because the 301 (and now 302's as we learned) pass on this value.
Hopefully that makes sense :)
-Dan
Hi evolvingSEO :-)
In your comment I see:
Update: 7/26/12 Hootsuite has changed ow.li from a 302 redirect to a 301 redirect. Thanks, Hootsuite, for reading! (Link to comment from Matt)
I think that you wanted to write owl.li instead ow.li.
So now, is better to use ow.ly or owl.li thinking about 301 redirects?
Thanks for your response.
P.D.: Sorry for my English :-) I'm Spanish and I'm improving it!
It really amazes me just how many redirects some affiliate links and/or URL shorteners actually go through.
BTW, if users don't want to use a browser plugin to detect redirects, they can use Redirect Detective.
Wheregoes.com is a similar service that I've used.
And herein lie the dangers of people who think they know about redirects and the dangers of them not being done properly. Although I've only used Bit.ly or Su.pr (StumbleUpon's version) the principles are still the same and I will be copying the general pointers, as you explain them much simpler than I do :-)
Oh and thanks for the heads up for the Chrome extension, always great to find another tool to make life easier!
Thanks.
Like the infamous penguin update has grabbed wpmu.org for the low quality backlinks from the templates designed by them. Isn't it good to add a 302 redirected link at the bottom, so that google doesn't consider its value in backlink profile. Will it help?
Your'e right, but why not 301 and get this over with?
Just adding a comment so folks notice the update.
Hootsuite has read the post and changed their owl.li redirects from 302 to 301. Thanks guys for your attentiveness and commitment to helping to provide the best user experience for everyone!
Wow, that's bizarre. I wonder how many black hat strategies use this to scam the allgorythm.
try searching for 'payday loans' in the UK.
there is a site fast-payday4vip.co.uk which is 2nd using a similar technique, doing something really odd with 301 and 302 redirects.
on page 2 for 'payday loans' is watcherofweasels.org and galorath.com. neither is about payday loans but they have been compromised in some way to 301 or 302 to fast-payday4vip.co.uk. the homepage doesn't redirect currently, so it either a) used to or b) just deeper pages redirect.
in my view there is definitely a bug with google where with the right combination of 301 and 302 redirects you can get a new site to rank highly, and as a byproduct it also causes the redirecting site to rank highly for the new sites keywords.
Good post, very interesting stuff !
Had some really bad experience with redirect chaining myself (much longer chains then just 2 "hops").
Didn't even think about looking at owl.li redirects, now I`m glad that I`ve choose the (.ly) version.
Will spread the word.
Job well done! I like the info so much.. I will not try to use url shortener anymore..thanks!
Impressive case, Shurelock! Thanks for the URL shout out in the redirect images :) - I always use bitly and never thought much about the transformation process and the repercussions on search.. great topic and I like how you show it all started with Rand's observation - why the community rocks socks.
"You should ALWAYS internally link to the current version of your URLs" - thanks for that reminder.
I like dgolds suggestion above perhaps this creates the impetus for someone to engineer a shortener which would maintain branding, is more consistent, and better serves search. #Mozcon!
A minor point, but I'm afraid we can't count on Googlebot continuing to crawl and index the original location of a 302 redirect. My company recently set up a 302 from the home page of a subdomain (call it a.site.com) to another (b.site.com) without bothering to tell me.
Less than two weeks after the change was put in place, a search on [site:a.site.com] no longer included the home page, and [cache:a.site.com] not only displays a copy of b.site.com, but the text above the copy of the page reads, "This is Google's cache of b.site.com."
Woah I think my head just exploded, but that's really curious as well!
Excellent presentation! Thanks for breaking things up with the videos. Bit.ly it is, with some investigation following your methods first! or perhaps just our own little shortener to keep things branded.
The conclusion here is that when you use 302 redirect Google indexes content from the final URL (original source) but displays the first URL (short URL). This is how 302 redirects are defined by Google.
Because in our case short URL was used in many places (twitter, blogs,...) - Google decided to use it as a preferred URL for this content because of stronger ranking signals.
Awesome post and investigation. I reckon google will take a deeper look into this after seeing that Bing is doing a good work.
Could always use another URL shortener that doesn't use 302 redirects..... https://www.michaelcropper.co.uk/2011/07/url-shorteners-for-seo-238.html
I loved the post, I even checked the results in google australia, check my results out https://yfrog.com/z/nfozgp
I think ow.li has been very smart about this. They provide a free service and they are getting the link juice from it. Whats wrong with that :-D
Never use some else's link shortener especially Bit.ly's
If you want to shorten and track links with it, use your own shortener hosted on your own server
one of our clients (actix) is getting outranked for an absolute blank page (how does this happen) for the keyword "radioplan"
https://www.google.com/webhp?sourceid=chrome-instant&ie=UTF-8#hl=en&output=search&sclient=psy-ab&q=radioplan&oq=&gs_l=&pbx=1&fp=ccd452f04fdfaa98&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.,cf.osb&biw=1920&bih=955
Dan - Great technical detective work here, I especially enjoy that you used hard references and critical thinking (very Shure-lock of you) at every turn possible. Cheers!
Great stuff Dan. My guess is that owl.li could easily pull the .ly domain out of the chain if need be during any further civil unrest (assuming they are backing up the data) - seems that the main thing from a control standpoint was to get the starting point to be the .li domain, then they can always do anything down the road, so that doesn't bother me so much.
That 302 is pretty vexing though; easy enough for them to fix I would presume, should be just a one-liner somewhere.
Caveat emptor!
Hi,
I don't know How Owl.li Is Ranking Above Own Page,
But i find here some really great resources in here for How URL Shorteners Work. Thanks for sharing great research.
A Great post agaain.. uffff