Last week Will blogged about what he called the World Series Spidering Problem where he looked at geo-delivery and when this crosses the line into cloaking.
The short answer is that showing different content to UK based users would be seen as cloaking, as it would break the "golden rule," namely:
Googlebot should see the same content a typical user from the same IP address would see.
I'd thought I'd use this and delve into an example showing some of the pitfalls. This post may get fairly technical at points, so apologies upfront for that. I hope that everyone will be able to take something away from this post. In order to take a look at how the US-centric view taken by Google can affect how a site performs in google.co.uk, we will take a look at a site a few of you will have heard of before.
Apple is often used as an example of a major brand that has gone down the sub-folder route to solve the issues surrounded by geo-location. There are 3 routes that you can go down to indicate the geo-location of your site:
- Using local tlds. This is the route that Amazon has taken, with www.amazon.com, www.amazon.co.uk, www.amazon.fr, etc.
- Using subdomains. Wikipedia is a good example (en.wikipedia.org, fr.wikipedia.org). Note Wikipedia and many others do the split at a language level, rather than country (en.wikipedia is used in the UK, US,and Australia).
- Using subfolders. This is the route that Apple has taken, with www.apple.com, www.apple.com/uk, and www.apple.com/fr.
Further discussion of the pros and cons of each of these routes will form part of another blog post in due course, so for now I'm going to go all techie and talk about the issues we see browsing from our little island across the pond.
The first thing to note is a common issue, which drives me mad. Despite being in the UK and Apple having a UK landing page, the sheer domain weight of the apple.com homepage means it is the homepage that ranks. This is a common issue, which is true for Apple and Amazon, and drives me craaaazy. Quite often it is only when I get to adding a product that I notice the price is in dollars and not in pounds. Sharing a language means pages targeting the US often look quite similar to pages targeting the UK.
Here are the results for a search for Apple from the UK on google.co.uk:
As an aside, there is a parameter (gl) that you append to Google results to set your geo-location, the idea being that if you append gl=UK, you see the results as if you were in the UK. When I tried this on a search for Apple, it removed the most relevant result!
Anyway, I digress. The observant among you will notice that there are a couple of extra checkboxes below the search box. Over here on our island we can choose to search "the web," or we can search "pages from the UK." For those of you have nodded off, pay attention because this is where it gets interesting. Watch what happens to the world's 33rd biggest brand (according to interbrand) when you check the "pages from the uk" option:
Scared? Confused? LOL? To remove any doubt, please note, Apple (the world's 33rd biggest brand) doesn't own apple.co.uk. To save you some time, please note that apple.com doesn't rank on page 2, or page 3; in fact, apple.com does not rank at all. No matter how far down you look, you will not find apple.com. Don't believe me? Try this search.
https://www.google.co.uk/search?&q=site%3Aapple.com&meta=cr%3DcountryUK%7CcountryGBYou'll see discussions.apple.com, sometimes we have seen images.apple.com, but plain old www.apple.com, not so much. Nada, Zip, Zilch.
According to a recent Hitwise study, around 14% of searchers use the "pages from the uk" option. This means that Apple isn't appearing in the results for around 14% of queries for their own name. As an SEO, branded search is your bread and butter, so something is seriously amiss here.
A bit of background: the "pages from the uk" checkbox documentation is here, and the key message is:
keep in mind that our crawlers identify the country that corresponds to a site by factors such as the physical location at which the site is hosted, the site's IP address, and its domain restrict.I have to admit that I don't really understand what a domain restrict is, but the take home message is that "pages from the UK" looks at the physical location of the hosting and the IP address of the site. I'm going to assume that they use the IP address to determine the physical location, which makes the IP address of your hosting crucial if you want to get the last 14% of your branded traffic, by appearing in the "pages from the UK" set of results.
When we first looked at why apple.com wasn't ranking when the "pages from the UK" option is checked, it drove us mad. As far as we could see, the website was hosted in the UK. The DNS queries we ran returned UK-based IP addresses, yet apple.com wasn't appearing in the results.
The answer lies in Will's post last week where he discusses how Googlebot always spiders from the US. When we tried querying the DNS from the US, we were getting ip addresses based in the US. Hence, when Googlebot crawled the site it was seeing site hosted in the US.
The reason for this change in IP address based on where you query from is due to Apple's server setup. Apple uses a company called Akamai, which does far more than just host the website. Akamai has a distributed set of servers around the world, which provide load balancing to ensure the site is always responsive. As Akamai puts it:
The Akamai EdgePlatform comprises 34000 servers deployed in 70 countries that continually monitor the Internet – traffic, trouble spots and overall conditions. We use that information to intelligently optimize routes and replicate content for faster, more reliable delivery.From Apple's point of view, this means (as far as I understand it) that there is a copy of their website on multiple servers all around the world. Apple's DNS is dynamically served by Akamai. Akamai dynamically changes the routing to ensure your request is dealt with as quickly as possible. This is slightly different to traditional load balancing in that Akamai isn't trying to (necessarily) return the server with most capacity. They are trying to return the route with the least latency, which is a combination of server capacity and network capacity on route to the server.
There is a good article looking at the Akamai algorithms in a bit more depth over at Wired. The key point is:
[The Akamai servers] keep in constant contact with each other all over the map, speaking their own special dialect of Linux. Each region has one mapping server and one or more content servers. All content servers, no matter where they are, are eligible to serve any content. The mapping servers monitor the local state of the network: How fast are the current connections to neighboring regions? Which connections seem to have gone down completely? They figure out which servers should carry which files, and then how to evenly distribute the hits for a requested file among the servers that carry it.The Akamai setup means when you are in the US and query looking for where Apple is hosted, Akamai returns an IP address of a server which is physically located in the US. Despite the fact that there is a server hosting Apple's website located somewhere in the UK, the fact that googlebot only spiders from the US means that it only ever sees the Apple site as being physically hosted in the US, and as such it doesn't trip the "pages from the UK" filter.
Unfortunately, there are very few options available that can push you into the "pages from the UK" search without adversely affecting your other results.
The most obvious option would be to switch your hosting to a UK-based host in order to get a UK-based IP address. This is likely to cause all sorts of issues for your US rankings, so it isn't advised.
If you have gone down the subdomain route, it is possible to have one subdomain hosted in the UK whilst leaving the remainder of your site hosted in the US. I don't know whether this would work, and I don't know of a way of doing this at a sub-folder level, though I'm not a DNS geek.
With both subdomains and folders you can set the geo-location from within Webmaster Central. This should help to flip the switch and let you appear in the "pages from the UK" results. Either Apple hasn't set this flag, or the flag alone doesn't flip the switch that lets you appear in the "pages from the UK" results.
With no sensible suggestions left, I can only suggest jumping up and down and making lots of noise about how the "pages from the UK" is very badly specified. How many users check that button thinking, "Being the patriotic person I am, I really only want to see results that are using UK based servers"? Surely a better specification for "pages from the UK" would be to return those pages that are targeting UK visitors. I hardly ever use the checkbox, but when I do it's because the results I'm seeing are too US focused and I'd quite like a UK perspective, or UK prices. I couldn't give two hoots about where the server hosting the website is located. This leads to the conclusion that for global companies the physical location of the hosting is not a particularly great indicator to their geo-location and the countries they are targeting.
The sheer volume of links to apple.com will almost certainly mean that this is such a small issue it's hardly worth the time it's taken to type this sentence, but I couldn't finish the post without mentioning the duplicate content that the Akamai cleverness introduces. www.apple.com is a CNAME to www.apple.com.akadns.net, meaning both www.apple.com and www.apple.com.akadns.net resolve to the same content.
**Update: Just after I had written this post, I noticed that ikea.com has solved a number of these issues, in that they use the Akamai services but also appear in the "pages from the UK" search. They have done a number of things well, though there are issues with their approach as well. I suggest if you are interested, you take a glance, and we may make it the subject of a future blog post**
I hasten to add, that unless they are already your clients, Apple have just saved a quite a lot of cash in SEO consultancy...
Well, I assume you can help Google by pointing the country of your domain on Google Webmaster Tools -- it should solve the problem for sub domain strategy.
For TLD strategy, Google Webmaster Tools points that it use the TLD for this, so you don't have to point anything.
The problem arises when you do sub folder strategy -- which would be the best for any case. AFAIK, Google Webmaster Tools won't allow you to point a country for each subfolder, which is lame. And this would be the best solution, I think.
You can also set this at a subfolder level. Check out this whiteboard friday post with Will from SMX West
As far as I know, setting geo preferences in Webmaster Tools only affects ‘pages from country’ searches, am I right?
yes Ann, I think you are right on this one.
Hi Ann, unfortunately, we have heard both directions of this and there is a lot of rumour and innuendo at the moment - not least, I think because how G works *at the moment* is not necessarily how it is going to work in the future... I think they have plans for it to do a lot more and if you are going to set it, you have to be prepared for that.
hi dudup, let me add here that it is possible to point subfolders to their respective targeted countries... if say you have mysite.com/italy you have to make sure that you open an account on GWT for www.mysite.com/italy and not www.mysite.com so you would have to open one account on GWT for each subfolder within the specified domain. cheers
david
Great post Duncan - really good. Please post more, either here or over at Distilled.
I have some contacts at Apple and will be forwarding this post over to them first thing tomorrow morning.
Now if only there was a really good answer for these big, multi-national companies that solved all the problems....
Duncan, excellent article, and one that highlights the need for Google to send spiders from it's own tld locations.
In regards to DNS, to point a subdomain at a host in the UK you'd use an A-Record, and point that new subdomain at an unique IP for your hosted UK content.
Thanks. I'm right in thinking though that you can't do the same at a folder level. There simply isn't a way of adding a DNS record to just apple.com/uk or similar? Is that right?
Excellent post. There are quite a few sites that have these problems.
There's no one easy solution for many of them.
Generally like the advice, but a few things.
First, the country code to generate UK results is GB, not UK. There's an entire list here, and we explained more about this in a recent article at SEL.
Since I'm in the UK, I can see exactly what someone in the UK sees -- and nope, apple.com/uk doesn't disappear in the way you simulated. Nor does it disappear if I force simulate using GB. Using the incorrect UK code is mucking up something, but that's not what what happen to virtually anyone.
Bigger issue, of course, that if I do select "pages from the UK," the .com goes away. Big problem, yep. Though few actually use narrowing like this, still big problem.
To go back to your tips, it sounds like people might only be able to do one of them. I'd say do as many as possible. IE:
1) Get the TLD for the country you want
2) Host that domain in the actual country, if you can
3) Get links from sites that are specific to the country you want to be associated with.
The first two are entirely in your control; the third takes more work. All have been recommended by the search engines for years.
The most important new change is the ability in Google Webmaster Central to associate a country with your domain. Haven't played with this recently to see if multiple countries can be tied to one domain. Also get mixed reports on how well it really works. But it should be on the list of things to do -- at least to help with Google.
As for putting things in subfolders/subdirectories, that has nothing to do with geolocation. I just don't see it as a geolocation tactic. There's no particular reason a search engine is going to say "ah ha, i see /uk, therefore this must be the UK section."
It can be part of an overall strategy, of course. Setup a separate domain for your country site. Then have links from that lead into a subdirectory of your main site if you want, if that's easier from an site maintenance standpoint.
Anyway, my thoughts. Interesting to hear more from others.
Hi Danny. Thanks for your comments
I'm feeling a little nervous since I think I'm about to disagree with you, and I never thought I'd have to do that!
I think the gl=uk parameter is correct. The following documentation talks about the gl parameter (admittedly for custom search), which links here where it says to use uk for United Kingdom. Matt also talks about that here
I'm also in the UK and still see apple.com/uk disappear (with gl=uk, for gl=gb I see apple.com/uk second)
I completely agree that you should do as many of the tips as possible, though in apple's case they don't own apple.co.uk!
The point about subfolders and subdomains is that webmaster central allows you to set the country flag at either a folder or a subdomain level. It can therefore be a fantastic geo-location tactic, since you can specify that /fr is targeting France for example.
Google has been a little mixed up in their internal use of the UK and GB nomenclature.
For a long time, a local .org.uk site showed up in WMT with this message:
"Your site's domain (.GB) is already associated with the country/region: United Kingdom"
They changed it a few weeks ago to read:
"Your site's domain (.UK) is already associated with the country/region: United Kingdom"
I would like to read more about the ikea example you mentioned. I like the UK series and I am waiting for more!
Hey thanks Duncan, this is a great post. Bringing Akami into this is something I may never have considered, but now it certainly makes sense.
I'd also been wondering for a week or two now how to view a google query from a different location. Here in this post you've supplied an answer, and I can't wait to look deeper into this and other parameters in the search URLs.
As far as what to do about this issue, I'd say that Apple should contact Google and find out what they say. If Google offers no solution, Apple's representative should ask if this would be an appropriate exception to the cloaking policy, as well as notify them of their intent. If Google were to know, they may add an exception rule or inform the representative that it is against their policy to do so. In either case, Apple can then make a more informed decision.
The Google Global firefox plugin is really handy for viewing local google results (though note that the gl= flag is not perfect as Duncan comments in the post).
Fantastic post, and many, many thanks for the gl= tip.
I'm in the opposite situation here, in that we have a .com site which principally operates in the UK. It's hosted in the UK, and the UK content is at the root of the site. We do, however, offer services to the US, located through a subfolder.
I've used webmaster tools to geo-target our US pages to the US, and the rest of the site to the UK.
The key search phrase for our US pages is different from the UK, thanks to differences in terminology, and of course the pages are optimised for this.
Conducting the US search on google.com from the UK puts our US page at position #11. The "pages from the UK" switch isn't present (since it's google.com, not google.co.uk).
Adding gl=US, however, our US page goes down to position #207 or thereabouts.
Changing this to gl=CA, we're back up to position #21.
Is it possible that google.com, in the US, might be operating an implicit "pages from the US" switch that trumps the setting in webmaster tools?
(PS: Gosh, it was hard to articulate that without mentioning actual URLs or search phrases! As a new poster, may I respectfully ask how specific I can be without crossing the line?)
Fantastic post Duncan. I really hope you post more tech based content in the future. Its a great coincidence that I just started looking into the world of CDNs. This really compliments all of the other reading I have been doing. Thanks!
So the thing that's been bugging me ever since we talked about it in the office this afternoon when Duncan was writing this post is: have akamai considered cloaking for googlebot in terms of which server they send it to - in order to avoid occasionally being seen to be hosting in the UK / Belgium / small African country just because that was the place that had low load when the spider arrived? Would they consider this in the future - i.e. as IP address location becomes more important, might they see it as important to present a consistent hosting location to googlebot?
And would cloaking your dynamic DNS be against guidelines?
Ideally Google should have a location-spider (loco-spider?) that spiders from a UP in sync with their own tld's.
For Akamai, I think they could set up as a default a mapping or geo-located gateway server that delivers the first matching location-based instance of a request, and then server the rest of the content from where ever in the ether has bandwidth. This would be a bottle neck of sorts and it seems to contrast with the mantra of speed in end-user deliverability, but would help with the geo thing. Meh... too much work. Just petition Google. I hear they listen to feed back ;D
You're close on that one, Will (or I've misread your post) - why not load balance across international IP's when the server detects an ordinary user agent at an unknown IP (let's say, a user with IE somewhere in Surrey). But, when a known crawler from a known IP range comes in, the website is always served at the correct IP location based on the requesting URL eg apple.com/uk is always served from the UK..
Very nice point Rich
Are you talking about language negotiation? My experience with serving the language based on the user agent is with a site hosted completely in Germany, Language is specified in html with <html lang="es" xml:lang="es">but languages are (unfortunately) not specified in subdomain or folder. Searches in Google.es for "company name" returns the companies main English site. When searching for "company name" in Google.es, pages in spanish, Google shows the spanish homepage of this company.
So this method does not work properly either in google. Since the language is not specified in folder or subdomain, Webmaster Tools cannot be used to set the Geographic location.
Great read this thread.
I manage a bilingual .com site, hosted in the my main market country, defaulting to my main market language and note dramatic SERP differences when a google search is conducted when &hl= parameter comes into play.
No matter how much i have set the webmaster central geo settings, the html declaration, and html lang setting, even the locality meta tag, i still underrank, and drop 8 places, for natural language query for my site when i select the "paginas de españa" (and those ranking above me display .com and not .es ending)
I suspect Dannys comment about the language of your most influential IBLinks is a factor. Ideally the ibl to your site should point to the url with the language parameter set in it i guess, but what happens when you cannot control in the ibl parameter and it comes from a english anchor link into a spanish homepage for example - big spanner or unimportant?
My question to others grappling with multingual matters is:
How critical can mixed languages in your homepage title tag be? eg. a french company name with an english slogan in the title tag for your homepag. Is that a spanner in the works for search engines or a surmountable distraction in determining language?
I noticed Ikea is including a language meta tag where apple is not.<meta name="language" scheme="rfc1766" content="en-GB" />
I think the syntax of this tag is invalid, but when I inspect the response headers I see the Content-Language coming through correctly...
Content-Language: en-GB
Maybe that's enough right there for Google to include a site in its uk results.
Oh! and you could dynamically serve the language= tag in the meta based on crawler requested url.. (see my previous comments)
Really great post.
Out of curiosity, when you search google.co.uk in the UK (but use the search the web option) are the results identical to those on google.com in the UK?
Does google.com redirect to google.co.uk in the UK? I think Rand posted something similar to this a ways back, but I can't find it...
No, they're not!
To extend my comment above as an example, if you search on our US keywords on google.co.uk using either "the web" or "pages from the UK", our US page is at position #1.
" As an aside, there is a parameter (gl) that you append to Google results to set your geo-location, the idea being that if you append gl=UK, you see the results as if you were in the UK. When I tried this on a search for Apple, it removed the most relevant result! "
Can some one please tell me, how can I set this parameter gl=uk on my browser?
It's part of the URL. Conduct a search on Google and then just add &gl=UK to the end of the URL in the address bar of your browser. Don't forget the '&', since that's what separates parameters in a URL.
Oh.. that was simple. Thanks dude !
Excellent Post Duncan!I have a doubt, does this issue arise only due to the Akamai algorithm? Is Google doing a good job otherwise?
Wow. Amazing post. I really didn't believe you and that link helped out.
Wow. 33rd largest brand and THEY DON'T SHOW UP in UK Google search.
How could Eric Schmidt allow this? (He is on the board of both Google and Apple...)
I think Apple's best choice is to use subdomains because they are treated as separate websites. I'm no expert but this would be my best suggestion for this MAJOR issue.
From Matt Cutts:
"Geo targeting by itself is not cloaking under Google’s guidelines because you’re saying "Take the ip address. Oh, you're from Canada (or Germany, or whatever). We’ll show you this particular page."
See more (and a link to the source) from some site called seomoz ;
I really like to hear more about Ikea! My compagny is hosted in Akamai too and we have to deal with these kind of problems.
Nice post .. keep the good work up !!
Fantastic post Duncan - techie but written in plain English so even I could understand it.
The confusion really does seem to come only when there are different versions of a site.
I used to work for a company whose site was a .co.uk but hosted in the US and we had no problems ranking 1st for 'search pages from the UK' or 'the web'. It would be interesting to see how it would have worked if we'd owned .com and .co.uk - I think then problems would have occured.
Looking forward to future posts :)
Great post.
I am in Australia and I own a site hosted in the US (dont we all - for you US'ers, dont *ever* complain about the cost of bandwidth). Interestingly when I search for 'skylines' without checking the 'pages from Australia' box, I get the same result as if I had checked the box (my site comes up at #3). I use the 'pages from Australia' box all the time. I do notice a difference with some results and have noticed that .com.au domains do not appear in some 'pages from Australia' searches.
Its harder for us Aussies to tell the difference between currencies when searching to buy something, too. :)
But at the moment it's pretty much 1:1 anyway isn't it? I was gutted when I went to Sydney and the exchange rate had slipped from Aus$2.5:£1 to Aus$2:£1!
See ISO 4217 for currency conventions you should be using on your website.
Really good stuff, Duncan. It's ludicrous that apple.com/uk shouldn't appear in a search "from the UK." The same thing seems to be happeneing with google.co.au, which shows this SERP for the web and this for Australian-results-only. The .com/au subfolder disappears as well, with asia.apple.com showing up in first position. asia.apple.com doesn't rank in .com.au without the national filter and has no snippet. What's going on there?
Great that you've also highlighted how certain procedures, such as Akamai's making sure the site loads quickly for users worldwide - can have less desirable effects for search.
Lots of great info here regarding geo-location, and thank you for that 'gl=' tidbit!
In case of apple.com/uk being outranked by apple.com, I reckon it might be because of the extreme domain authority of apple.com and heavier linkage of UK based sites to apple.com(total 2642849 backlinks) as compared to apple.com/uk(28335 backlinks).
The technology that Akamai uses sounds really interesting but if Akamai serves different Server IP locations for sake of load balancing, don't you think that the results to the query should be slightly more varied when made from different locations. I mean different results at different instances made from different locations.
BTW, it was a really useful post Duncan. Good Work. Cheers !
That's a great post Duncan, geolocalization is always a big problem.
I had the same problem with an international e-commerce: we choose a multi-ccTLDdomain strategy ( like amazon ) and only with the right use of meta language we solved the problem of geolocalization.
Otherwise, there's another problem if u choose a multi-domain architecture (like amazon): the single sign-on. If u have a multi domain site with an area restricted by login and password you can't use cookie for grant access in all countries. (Cookie is valid only for one domain) So, you need a single sign-on system = more work for developers = more chance to use this strategy beacause is more expensive.
good post Duncan.
"With both subdomains and folders you can set the geo-location from within Webmaster Central. This should help to flip the switch and let you appear in the "pages from the UK" results. Either Apple hasn't set this flag, or the flag alone doesn't flip the switch that lets you appear in the "pages from the UK" results. "
if you look at the source code for apple.com you will not find the meta in the head, that could tell you that they are registered at the Webmaster Tools. So they did not mention Google, which subfolder goes to which country !
You can verify with webmaster central by uploading a file, so this doesn't necessarily prove it.
that's right Duncan! I forgot that you can do it as well by adding another html page. But I guess they would have loaded their sitemap at the same time, don't you think so !? but they haven't got any...
www.apple.com/sitemap.xml
Eric Schmidt is a member of both Google and Apple boards and they don't need sitemaps, webmaster centrals or any of these. If they wanted to be no. 1 in Google in the UK, they would be. Why they are not? Because Chuck Norris said so.
Sorry, guys. Too much work today :)
thank you for the widget Mighty. I'll see if Chuck gets angry while I am writting too much stuff, without really looking at them...especially when its about apple..
They have a 'standard' sitemap: <link rel="index" href="https://www.apple.com/uk/sitemap/">
*** you will not find the meta in the head ***
Is it defined in the HTTP header instead?
Great post there - a real eye-opener for those trying to geo-target with complex systems.
Don't forget .co.uk TLD's! They appear in the 'pages from the UK' .co.uk search too even if hosted in another country (however there can be exceptions about which country they rank in :(), I'm guessing that the TLD is the 'domain restrict' according to Google...
Excellent, excellent, excellent post. I'm located in Rome and we have the same issues with all of our markets around the world. Google indeed, ranks agencies and other websites that metion our brand in all it's different country flavors. .DE .SE .ES. .PT so on and so fort. This is the stuff we need to focus our energy. SEO is INTERNATIONAL. I would love to see more post like this. Congratulations dunks!