Not everyone realizes it, but Google has been serving different search results to mobile phones than desktop computers for a long time. Beyond that, they serve different results based solely on the handset you are using to search. The differences are often subtle, or focused on the order of Universal Results that are included in the mobile result-set, but Google is algorithmically trying to prioritize content that will work well on the phone that submitted the query, and give less priority to content that might not work on the phone. If you want to compare for yourself, MobileMoxie has a mobile search simulator tool that allows you see the results of one search query across three different phones at a time (cool!)
Now Google has launched a new smartphone crawler, and this will likely push the differences between desktop search and mobile search more into the mind of the average SEO. (Think: “Does Google consider this a smartphone or a feature phone or something else? I don’t know!”)
So let’s see if we can demystify the impact of this new smartphone crawler using some data. This is the beginning of a three-part blog post series, all focused on the new Google smartphone bot. This first post will focus on how it works and what sites will be effected. The next post will focus on how you can optimize your mobile content for the new smartphone bot best by creating effective mobile redirects. The final post will discuss common mis-indexing problems that mobile websites have and review how to protect your mobile content from search engine indexing problems.
Mongoose Metrics published a study that broke down how the top sites in the US are handling their mobile traffic. They divided the QuantCast Top Million sites (most US traffic’ d sites) into 3 categories based on how they were publishing mobile pages: Server-side redirection, JavaScript redirection and what they call ‘cloaking’ or selective serving of HTML assets. The study went deeper, and looked at results for iPhone, Android and RIM requests, but we can just look at the summary for all smartphones, which show the following stats:
For SEO, these results are each unique and relevant; here is why:
Server-Side Redirection
This is a mobile site-architecture strategy that uses two (or more) urls; one for the desktop or primary content, and one for the mobile content; designations can be added for tablets, WAP and other devices as well. The mobile content can be on a mobile subdomain, a mobile subdirectory or a totally different domain, and those decisions can all impact the content’s ability to rank. The mobile urls can be static and optimized urls or they can be temporary dynamic urls, which are usually stuffed with the exact paramaters of the mobile page request.
Until Google’s new smartphone crawler, this mobile SEO strategy relied on the rankings of the desktop pages which automatically redirect to mobile content when requested by a mobile phone OR building independent SEO value for the mobile pages so that they would rank on their own merit. This strategy sometimes also includes joining the mobile pages with their desktop counterparts by using the canonical tag to help share SEO value. What is most relevant for SEO is that that both versions of the page are left crawlable by search engines. This might be important if you target lots of WAP phone searches, which are still sometimes using a separate ‘mobile-only’ Google index and are not affected by the new smartphone crawler. This could also be important if there is a future algorithm shift that puts a stronger emphasis on mobile file size (which could still happen because it is so important for a good mobile user experience).
SUMMARY:
RESEARCH RESULTS FOR SERVER-SIDE REDIRECTION: 52.52% of the QuantCast Top Million Websites in the US may see an immediate user experience benefit from the new smartphone crawler. Once these sites have been crawled by the new smartphone crawler, they will be serving mobile content to mobile users from search results automatically, without their server having to process each of the redirect requests.
JavaScript Redirection
This mobile site-architecture strategy also involves a primary and a mobile url for each page and an on-page JavaScript redirect is included from the desktop page to its mobile counterpart. The strategy actually relies on the FAILURE of the search engines to crawl and execute JavaScript in order to work properly. Frequently this strategy relies 100% on the desktop page to rank well in smartphone search results, and the mobile pages are blocked from search engine indexing in the robots.txt file, to prevent the risk of duplication or confusion in the index. In some cases, this JavaScript is detecting specific phones, and redirecting to landing pages that are just built for those specific phones, which can get very involved, but is great for user experience. Unless the new smartphone crawler begins to execute JavaScript redirects (unlikely), sites that rely on this method will not benefit from Google’s new smartphone crawler.
SUMMARY:
RESEARCH RESULTS FOR JAVASCRIPT REDIRECTION: 2.15% of the top million websites are not going to benefit at all from the new smartphone crawler, but may have already had good results in mobile search rankings without indexing mobile-specific pages.
Cloaking or Dynamic Serving
This mobile site architecture strategy relies exclusively on one url that can display a page with different characteristics, depending on the device that requests the page. What content is served is determined by the server and something that is usually described as a ‘mobilization engine’ or a ‘transcoder.’ These are essentially databases of rules and content at various sizes or stages of degradation that can be sent, depending on the capabilities of the phone requesting the page. With this system, a desktop computer will get the full version of the site, but a mobile phone might be served a similar HTML skin, with smaller components switched in, to replace place of the larger elements that are served to the desktop computer, all on the same url.
A similar but less sophisticated version of this type of mobile publishing can be accomplished using mobile-specific style sheets and media queries to re-render or re-organize the content on the page based on the screen size of the device that it is requesting it (Responsive Design). This strategy uses only one url which might be appealing if you are trying to keep things simple for maintenance or SEO reasons. In both cases, bots will be served content based on the device or browser that they are emulating, so the smartphone crawler would likely (hopefully!) be served a smartphone-friendly version of the page unless the JavaScript is purposely made un-crawlable.
SUMMARY:
RESEARCH RESULT FOR DYNAMIC SERVING/CLOAKING: 45.33% of the QuantCast Top Million websites might be at a disadvantage in terms of bandwidth, user-experience and load-time. All of the dynamic processing is still required by their server to render the page each time there is a mobile page request. They will not likely benefit from Google’s new smartphone crawler, and might now be at a disadvantage, (in terms of the load time of the dynamically generated pages) where they probably had a slight advantage previously.
Mobile SEO is an ever-changing field. The search engines don’t all agree, and still don’t seem 100% sure how to best rank and evaluate mobile search results. Your best option is always to know how things work, keep a close eye on how your sites are ranking and do your own mobile testing wherever you can. Until the smartphone crawler many people were unaware that mobile search results were treated differently from phone to phone. Now you know that testing on your own phone might not be enough; you may have to address the new Google smartphone crawler with some well-planned mobile strategy. Hopefully you can use this analysis to help determine how your site will manage with mobile serving, and still please the new smartphone crawler.
Stay tuned for the next post in this three-post series about the new Google smartphone crawler. It will cover how you can optimize your mobile content for the new smartphone bot best by creating effective mobile redirects, then the final post in the series will review common mis-indexing problems on mobile and discuss how to update your mobile server settings to prevent search engine indexing problems.
I think you've misunderstoud Responsive design, at least a little bit. There isn't neccerarily any javascript involved, just simple CSS media queries fix issues with screen sizes. The whole point is to make the entire website scale to any screen, whether it is mobile or a full hd tv.
This, also will be the way forward as it stops having to make duplicate content essentially which as we all know can be a huge problem!
Hi Leon - Thanks for the comment. It is quite common for developers to include JavaScript and jQuery when building a Responsive Design site. It is possible to do Responsive Design without it, but generally it is an important way that developers can improve usability and performance for their Responsive Design sites:
https://responsejs.com/
https://speckyboy.com/2011/10/24/25-jquery-plugins-to-help-with-responsive-layouts/
https://www.quirksmode.org/blog/archives/2011/09/responsive_desi.html
I agree with Leon that your section on "Cloaking and Dynamic Serving" doesn't really do justice at all to responsive design and claiming media queries as "less sophisticated" is somewhat apocryphal.
Whilst some javascript is often used to provide some features of a responsive design it certainly doesn't have to be the cornerstone of it and it's easy to make sites render well without javascript and "enhance" for those browsers that can manage it.
The links you've posted:
1. Response.js is a nice to have - it deals with resizing media and optional code blocks (although optional code blocks are, in my opinion, better served by showing/hiding them using CSS)
2. The speckyboy list has some tools for use with responsive layouts, mostly to do with gallery widgets and the like, none of the plugins on that page are necessary for a responsive layout to work, but they make the experience better in some cases
3. The quirksmode article emphasises degradation over enhancement and says that mobile first layouts are a thing of the future. I guess he was only talking about 7 months in the future because that's how we build our sites now.
Luke Wroblewski has written a good book on the subject and his blog is worth following if you're interested in the topic: https://www.lukew.com/ff/entry.asp?933
Regarding "load time" a mobile first responsive layout using "upwards" media queries is superior from the user's perspective to that of a redirected URL for obvious reasons. If Google crawls, indexes and serves the mobile subdomain then the user won't notice a difference when arriving at your site via Google which means that in terms of Google search engine results, server side browser sniffing is now on an even footing with mobile first responsive layout. Mobile first still wins when someone types a URL directly into their browser, though, or arriving from any other source apart from a Google search.
The only downside is that a Desktop or tablet browser will make an extra request for the CSS file(s) for it's own screensize. This is a reasonable trade-off in my opinion as it puts the pressure on the machine likely to have the highest bandwidth and processing power and serves the bare bones requirement to the mobile device which is undoubtedly on a slower connection/weaker processor.
Hi Cindy,
first of all I must tell you I'm very happy to see you again writing here on SEOmoz, as Mobile is no anymore an option, but a reality and probably the field of SEO we will need to engage ourselves most of our time.
Said that, as Leon pointed out, what wasn't clear to me reading your post is: what we should think about the relation between responsive design and Google? In fact, from the post, it seems that responsive design (or the definition given to it in the post) is not the best way to go in terms of SEO. If it is so, it seems somehow a contraddiction, as it is probably the easiest and more user friendly (both sides, back and frontend) way of having just a site prepared both for desktop and mobile browsers, with the advantage of having the same URLs.
But, maybe, I'm just confused.
Ciao
Hi!
The point was - Responsive Design is ok if you are careful to pay attention to the load time and usability issues that it causes, and the limitation that it puts on your design and layout, but Responsive Design sites will not really benefit from the new smartphone bot, because they don't use mobile redirects, which is what the new smartphone bot is lookinng for.
Redirecting to a mobile-only page is better if you want to have a more unique mobile page that can not be achived with Responsive Design, and if you want to take advantage of the new smartphone bot. There is not one answer that is best for every site - it depends on your goals.
" Responsive Design sites will not really benefit from the new smartphone bot," That makes no sense to me. The SmartPhone bot will crawl the mobile optimised content and realise that the the page-size is reduced and the experience optimised for mobile...
"45.33% of the QuantCast Top Million websites might be at a disadvantage in terms of bandwidth, user-experience and load-time. All of the dynamic processing is still required by their server to render the page each time there is a mobile page request. They will not likely benefit from Google’s new smartphone crawler, and might now be at a disadvantage, (in terms of the load time of the dynamically generated pages) where they probably had a slight advantage previously."
This doesn't seem to make any sense. A "server side redirect" as you're calling it passes back a 302 on the initial request and forces the client mobile device to make a second http request before serving content. By dynamically rendering your page you are sparing the client an entire extra round trip. All the dynamic processing required can still be cached just like any other dynamic page and let's face it -- who is rendering pages that aren't dynamic anymore. In my opinion as a developer of mobile content this method is far superior to the other two outlined so I'm not sure where the basis for this opinion comes from.
EDIT: Also, calling it "cloaking" still carries a bad connotation but I think Google has made it clear that this is perfectly fine for a practice like providing a good UX for the device requesting your pages. I'd hate to see SEO practitioners scared away by this label when it is the best approach IMO.
I think there are advantages and disadvantages to each of the options, and it really depends on what you are trying to achieve, and what kind of constraints you are working under. There are things that can slow any of the options down, depending on what and how you are loading stuff. You are very correct about appropriate cache settings -very important for user experience!
Great topic BTW -- I'm really glad you are bringing mobile search optimization into the forefront for discussion. I'm not sure who these downvoters are but I'm certainly not one of them.
I still don't see the advantage of redirects to a mobile url or ESPECIALLY javascript redirects vs. just rendering the url appropriately for the client device. I understand that Google is making strides to cache the 302s so that their users get the advantage of a faster experience, but what about direct traffic, viral traffic etc.
The fact that google is caching the url returned by the 302 is a great indicator that they believe the optimal experience is one that does not include an extra http request. I agree with them wholeheartedly but I think we as mobile content authors should provide that same fast experience to everyone, not just visitors that originate through google search results.
Javascript redirects are an even worse user experience -- especially on low bandwidth mobile browsers. Here you have to completely resolve a url, load some of the content, and then get a window.location to a new url that you then have to load.
Given these considerations DYNAMIC SERVING/CLOAKING is the clear winner IMO.
Great article and great comments.
I have one question though.
Wouldn't you always want a single page to have the highest amount of Google-juice instead of splitting it between two pages?
1) www.yoursite.com/myArticle
2) m.yoursite.com/myArticle
I mean, if you serve these pages to Google, sure it may index them properly for each device, but what about ranking signals - wouldn't they be split when users start sharing the content (linking to it from all kinds of places)?
If so, wouldn't it be better with responsive design or using www.yoursite.com/myArticle#m as a way of creating mobile pages, that doesn't split ranking signals?
According to what Google wrote, it seems that ranking factors for the smartphone bot are still determined by the desktop page, but the bot just slots in the mobile page url and potentially title tag and description. So if that continues to be the case and you wanted to consolodate value, you would be fine to canonical tag the smartphone page up to the desktop page. The only risk there would be if the canonical tag makes the mobile page drop out of the wap index - a risk if you rely on lots of wap traffic, but most sites don't worry much about feature phones any more.
So you agree with me, that the ranking signal (sounds as if it) is split?
If the ranking of the mobile content is determined primarily by the desktop version/url, then shared links to the mobile version of the page are not counted/used - which means they are wasted... unless you use the canonical tag and hope the search engines will honor it?
I'm a bit surprised that it took 35 comments for somebody to properly question the effects on the incoming link equity that this has. Well done Esben! :)
On top of the split link equity problem you have also got the potential duplicate content problem (same content on multiple URLs). These could be mitigated by the use of canonical tags and 302 redirects for Googlebots, however, you're still betting on Google "figuring it all out" properly.
The one URL for all thus got to be the preferred solution for SEO, as long as your back end can support it (make sure to set up automated tests which ensure that the main Googlebot sees the desktop version whilst the Googlebot-mobile gets served the mobile one).
In my opinion SEOs should never attempt to dictate site design for any project. There are many much more qualified people who should make these decisions. User interface designers, user experience designers, general web developers, etc should be given the freedom to operate as they see fit.
As an SEO your job is to work with these people to develop a product that is awesome, user-friendly, AND can be crawled, indexed, and ranked well in the search engines. If the design team decides that responsive design is the way to go, then you need to work with them to achieve your goals. It's not your place to convince them to go in another direction unless you can build a case that their decisions will be completly impossible to work with from an SEO standpoint.
Putting SEO above the design and usability of a website and getting in the way of your design team will only hurt your project in the long run. Stick to what you know and let experts in other fields do their jobs.
I disagree - I think SEO, design and development all have an equal stake in the matter. All three must work together effectively, without one or the other having less of a voice. All three are equally necessary for website success! Saying that SEO's analysis or opinions are only important if the result will be 'completely impossible to work with from an SEO standpoint' is not fair and would not produce the best result possible for the site.
So, is it possible for responsive design sites that Google starts looking for the mobile.css, and then gives responsive sites credit in the mobile SERPs?
Yes - it is definitely possible. Hopefully that is the case and you are getting credit for the handheld.css, but reading between the lines of the Google post, it seems like this new bot is mostly looking for mobile redirects. The smartphone bot will get whatever you would normally serve to an iPhone.
If you do your responsive design mobile first (with upwards media queries) this isn't an issue - the default is mobile.
Very nice post Cindy. However, I do have one technical question that pertains to usability. How do you execute a server-side redirection to detect mobile browsers, but also offer a "View Full Site" link in the footer of the mobile site to allow visitors to switch to the desktop version, and not put the visitor in an endless loop back to the mobile site? Thank you.
A common strategy is to set a cookie with the user's mobile preference. When you initially detect that they are mobile do can set a "mobile" cookie value to "true". Clicking the footer link to "View Full Site" sets the value to "false".
Before executing the server side redirect the application would check for the existence of the cookie. If the cookie exists, then the application would honor it (preventing the infinite loop), otherwise it would sniff for mobile, set the cookie base value and execute the redirect if necessary.
Just as a reference, Target.com sets this cookie to 7 days, which I think is a pretty good timeframe.
Also -- for our site we are planning to start showing "View mobile version" for those who clicked on "Full site" link and we've determined their user agent to be a mobile device.
Great idea about using the cookie. I appreciate your suggestion.
Server-Side Redirection "This is a mobile site-architecture strategy that uses two (or more) urls; one for the desktop or primary content, and one for the mobile content"
In this case, Does each computer and mobile device have one (identical) URL? Doest each URL (mobile and computer) get ranked/crawled seprately from a diffirent type of bot? (A bot that wants best user experience for mobile regardless of computer version)
In this case, does the content get indexed seprately? or ranked seprately? From an SEO point of view should I build relationships with relavent Mobile sites to link to mine? Are there any current mobile practices for SEO?
Its a very nice and catchy information for mobile content optimization. Mobile search is going to be the new search tecnology.
I am confused........
Sorry you are confused - please take a look at my response above. The mobile strategy that you choose really depends on your goals. The new smartphone bot will help some mobile sites, but not all of them.
I was a bit confused too honestly but reading the rest of the comments helped. Thanks! :)
It's a really interesting article and certainly gets you thinking, however, with regards to responsive design... well that's kind of a given really and usually it is reliant on a javascript query, coupled with a CSS stylesheet for mobile (at least that's how I do it).
Ok... so it's not good for SEO - but as firoelli1 pointed out this appears to be a contradiction in terms.
So any suggestions on what web developers could do instead of using this easy, straight-forward technique?
I didnt know Mobile search had different results than desktop. Also, iPhone is getting completly different results than BlackBerry as well.
Nice summary of the 25 page study!
My personal preference is to stick with the server side redirect, since now Google actually replaces the desktop version's title and meta desc with the mobile version's right in the SERP. Now that's a good user experience.
...or at least it's supposed to
To start with, I am not aware if Google has a prescribed set of guidelines for mobile webpages. However, we need to take into account certain factors here:-
So, for ranking better on Google, we will need to have customized Meta Descriptions & customized Microdata too. For this, a Javascript cloaking will not help to rank on a mobile Google SERP, even though it provides a better user experience, and saves on server resources.
A server-side redirect to a mobile sub-domain seems like the ideal set-up for ranking for mobile searches. Also, mobile searches demand instant information, and a person looking for information from a cell-phone is desperate and will have a shorter attention span. Here the importance of the mobile sub-domain increases. You cannot have multiple metas in the same HTML file on a server. The Google bot will be able to read both, but since the rendering of descriptions will be done through a script, there is a chance that it could perceive multiple descriptions as duplicate content.
Personally, I feel that it is best if we separate desktop & mobile versions of the HTML content, that too preferably through a sub-domain rather than having a sub-directory. In my short experience as a SEO professional, a lot of CMS setups attribute root directory features to sub-directories by default. You would not want that from a SEO perspective.
That's still a little hazy to be honest. I've trialled search on a number of Android, Blackberry and iPhone devices and actually receive equally good results for core-keywords on Google. Whilst we use dedicated mobile pages to redirect customers to the search results still perform better for our core standard pages than the mobile ones. Curiously enough we receive around 15% of all hits from mobile - the majority of this traffic goes directly to core pages rather than mobile.
You can use the argument that mobile search is directly impacted by the type of mobile website you utilise but this is a case in point where the argument clearly falls down.
Hi James. Thanks for the insight. I may have a possible explanation for the fact that your core pages are ranking on Mobile SERPs for most of your keywords.
I am assuming that when you say the core pages are ranking, you have checked out the URLs that appear on the SERP listings.
You have mentioned that you have dedicated mobile pages for mobile users. However, I checked out your website, but it was still showing me the core HTML url itself. (Please correct me, if I have gone wrong here itself!) If my observation is correct, your main website may be using CSS media queries to serve mobile optimized content to users. So a different CSS is pulled up whenever a mobile agent tries to access your site.
So whenever the Googlebot-Mobile tries to access your site, the URL must not be changing. Only the source HTML code refers a different CSS. So, in effect, (unless Google has the capability to understand that a different CSS is being pulled up & crawl that!) you are showing the same content to the mobile version of the Googlebot. No wonder, your core pages still rank.
Probably, your redirect rule, will require minor changes, to identify a mobile query. This could also be a potential reason for you getting only 15% of mobile traffic to the mobile version of your site.
Let me know your thoughts.
It depends which website you were looking at... I have around 4 different sites. The one I use the mobile pages for is https://www.iKubeinsurance.com. The mobile side was setup as a test to see if it made much difference - it doesn't. We still receive the bulk of our traffic to the core site pages.
Good point - How does the conversion rate and page views per visit compare when you serve the desktop site to the mobile phone vs sending the mobile site to the mobile phone? Usualy mobile visitors are more engaged when you send them to a mobile site, and more willing to convert too.
Google's definitely got mobile development guidelines. They've also done a few webmaster central blog posts about mobile development.