Team Fresh Egg has noticed a strange phenomenon, and it appears to be an ongoing issue from at least August 2006. It seems that home pages (and only home pages) of .com sites hosted in the UK are disappearing from Google's results when you do a site: search and filter the results to only show pages from the UK. I've played around with various sites, and while the home pages pop up under the normal web results, they mysteriously vanish when you add the UK-only filter.
The Google Search News forum Lee pointed me to mentions this problem, and several people have complained of fluctuating results (e.g. they're gone, they come back for a while, they're gone again). While duplicate content could be an issue for some, surely it can't serve as an explanation for all missing pages.
What do you think's going on here? Is it a Google glitch? Are they experimenting with various filters? Does Google have a "USA a-ok, but UK can kiss my arse" mentality? Have any of you experienced a similar phenomenon?
By: Rebecca Kelley
Home Pages Disappearing from Google's UK Results
Search Engines
The author's views are entirely his or her own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.
I can't see why any deliberate action would only affect the root domain, and none of the internal pages.
We ruled out Registrars, Server setups, Link relationships, and a dozen other things we tested. Everytime we thought "just maybe there's a pattern in..." we found sites that disproved the theory.
The trick to recognising this precise phenomenon is to run a site:www.example.com type search, first without the local filtering, then with, and the difference is that the root domain url, and only that root domain url, will drop out of the local results, leaving all sub-pages of the same domain listed. Obviously this makes no sense. If the domain is local, all its pages must be, and if the domain is not local, then none of its sub-pages can be.
A live example to illustrate (selected at random from the scores of sites we've been examining today): Exhibit A (web results) // Exhibit B (UK results)
How broken is that?
Weird. That site seems to have minimal link juice and trust. It also has loads of pages that are dead indicating its undergoing a few changes. Maybe one of the changes was a server move fom the US?
The main time I have seen pages dropping from the UK serps was when a site moved from the US to the UK in the last few months but some of the datacenters didn't seem to have caught up.
I have posted some data on UK Specific results and their importance to business on the Fresh Egg Blog.
UK Specific Search Results Data
I think the best way to compare is to take a co.uk domain hosted on a US (not UK) IP and we see that it ranks well in Google.co.uk and on the other hand let's take a .com or .net domain hosted on UK IP that ranks OK as well.
Now if we take the .co.uk domain and host it on UK IP it will fly by the .com or .net domains. So there I see an advantage for .co.uk domains.
A .com domain ranks at #1 on Google UK (UK pages only) for the most competitive phrase: search engine optimisation
If that works, but other .com domains wont rank for other keywords, it is a disparity not explained by just the .com / .co.uk geo location spec.
It was a time when .co.uk domains hosted on US (not UK) IPs weren’t ranking in Google.co.uk well. People have complained and I think Google is giving more credit to .co.uk domains regardless where they are hosted. But .com, .net, .info domains even if they are hosted on UK IPs because of that they will not get the same credit.
Also international inbound links only work if you have a .co.uk domain.
We all know that Google displays the most popular pages and they show in Google.co.uk inner pages of a .com or .net domain only if there are not many .co.uk domains competing for the term.
Google.co.uk results are good but they need to know that UK companies/ organizations/individuals preferred to register a .com or .net domain so we think .com or .net domains are better quality. Google understands that a .com or .net domain hosted on an UK IP is from UK but they do not have same credit as the .co.uk domains have.
The question is the .com, .net, .info UK owned domains and hosted where they found a good deal (not UK IP) will ever have the chance to show in Google.co.uk?
Mosxu,
I don't believe this to be a case of .com domains being treated in a bad way. This is a case of Google removing a group of index pages from its index.
Deeper linked pages from .com domains are still appearing in UK specific results as they were.
Whether this is intentional or a fault remains to be seen. Personally, I think Google has made a mistake here and I expect to see the results return to normal in the not to distant future.
The fact that this problem is occurring in such a critical time for some of the sites affected will force them to alter Christmas sales targets. But of course there is the safety blanket of having Adwords available :(
UK businesses that have a non .co.uk domain and who choose to get cheap hosting from outside the UK usually are not usually aware of the geographical filters.
The thought of saving a few pennies on hosting is all too appealing for some! But at what cost?
Google can be misterious.
Check out this link:
https://adwords.google.com/select/TrafficEstimatorSandbox
Who said Google never talks about a sandbox???
What we have noticed is that if the UK website is being hosted on a non UK server, and the domain ends with a .com, then the uk search thinks the .com is an international domain name.
That's the expected behavior indeed, if a TLD is international, like .com, the hosting determines in which local results a site should show up. So a .com domain hosted in the Netherlands should show up in the local Dutch results as well.
that's crazy! when are they ever gona get this thing right!
Hi, I am new to this forum.
I have also been noticing very strange things happening to quite a few sites, including 2 of our own sites since the beginning of October. Both of our sites are .co.uk domains, and are both hosted in the UK, but since the beginning of October the homepages of our sites dropped slightly in rank and had always been ranked in the 1st 10 or 20 results, also the subpages had very good rankings but these have dropped to a staggering 510. At first we thought maybe there was a problem with the sites but the same thing as also happened to some of our competitors. Also, the odd thing is that today the sites are not even showing at all on Google UK but are now shown on Google.com, does anyone have any idea what is happening? this is really frustrating and has already affected our revenue quite badly.
I buy a co.uk domain host in UK first. google crawed well. I change the host to USA after 1week, and the condition look well.Its show on google.co.uk but google webmastertool report that there are many links can't visits.What's wrong with it?I'm a new comer of website building. Any guy could give me some suggestion on the host or the seo? many thanks the site is buychinawholesale.co.uk
Matt Cutts has just issued a very useful infrastructure report which also covers this issue. Although it was thought that the disappearing homepage in uk filtered site: operator results was fixed in December 06, it seems that there still exists and issue which is being addressed. https://www.mattcutts.com/blog/infrastructure-...
This now seems to have been resolved: .com homepages reappear for Google UK searches
Actually, it's Not. Since yesterday we've seen 2 new sites that didn't used to have this problem now suddenly showing the problem, and it's still on now.
It's been more than 5 days since Matt Cutts posted, so they're either delayed on their final fix, or the final fix actually broke things that weren't broken previously!
We've found this occuring regularly and were advised to change some content. We did this and came out in UK results when we were spidered, but dissapeared two days later.
Is it a case of changing content every few days and try to beat the system?
Who advised you on this?
It doesnt make sense to me that it would be a case of changing content every few days, as if that were the case, why does it only affect the home pages of .com websites hosted in the UK, and not .co.uk and .com in the US and so on?
I am seeing this with .com and .org domains. The homepages have gone plus other pages when the uk filter is selected. I too have had it in the past for about a week, but it passed. It isn't affecting .co.uk domains on the same UK IP.
I thought they had sorted this problem but it's back with a vengeance!
Not only are all the sites that were disappearing gone now, but even ones which weren't previously affected have started being dropped from Google UK results, even though they ALL continue to rank perfectly fine for Web results, and all are still hosted physically in the UK (and resolve to UK IP addresses).
I have a .UK.COM domain. Previously i considered this slightly better than a .CO.UK domain since the UK had more prominence i thought better for searches which included 'UK' in the keyword/phrase/search term. If Google not considering .UK.COM a UK extension (probably because not an official Top Level Domain) then i think they presume its just a .com and have binned my home page from google UK. I still have all my google worldwide SERPS but UK specific SERPS I have help Page 1 first field of view for 5 years have now totally disapeared since beginning of october.
I've just been noticing the same thing myself of late. The UK specific filter treats various sites wildly differently - and I can't discern a pattern.
Some .coms appears just as they do either way round, whilst others get a real shoeing in the rankings. All our sites are on UK IPs - surely Google must know that UK companies can't always nail the .com version of their domain name. It's a bit screwy.
Someone, somewhere wondered if Google might be contemplating increased importance to .co.uk over .com irrespective of the IP address.
I'm hoping that isnt the case, as it could spell trouble for a lot of multinational companies that target UK and US and Europe and......
Maybe directly related or not, but Google definitally has something afoot.
https://www.wolf-howl.com/seo/more-google-craw... https://www.wolf-howl.com/seo/google-dropping-... https://www.johnon.com/158/banned-from-google....
Apples and oranges.
The discussion here is particular to just home pages being dropped from country specific listings, not entire sites being "banned" as seems to be the discussion in the links posted.
Sites getting banned could be due to a whole lot of reasons. This is much more likely to be a weird Google test or something amiss in their algo.
I've seen this on a bunch of .com sites hosted in the UK. It seems to vary across datacenters though and it always sorts itself out.
I'm seeing the same phenomen for some .com and .info sites hosted in the Netherlands on Google.nl in local Dutch results.
We've seen this happening to some sites we manage too. Definitely NOT an IP address issue, nor a duplicate content issue. Seems more like a random Google glitch.
Our take... sit and wait it out!
This happened to us as well. Our index.html at (theledsigncompany.com/index.html) dissapeared 3 days ago and is no longer cached. The root / is still showing at Google, as well as the same amount of pages that normally do. We are in the USA.
Thomas
Not really related to the issue at hand, as your site is not hosted in the UK nor has it a co.uk extenstion...
But what you're describing could well be related to duplicate content, as root / and index.html are the same pages.
My personal website also disappears from time to time from the Google SERPs.
I wonder if aliens are abducting our sites?
He did act rather strange the following days, and didn't want to talk about what happened...
Just to say I love the bit about alians on the web some thing strange seems to happen at times. Some times i think how easy it might be one day for some one to send explosives over the web.
Several of our sites are doing it. Also, they seem to drop more for UK only rather than UK worldwide. My pet theory is fiddling with the weight of the server location part of the algorithm.
It's definitely not a dup. content issue either. Interestingly though, I've been noticing it with Ask.com too. Anyone else seen that?
I know for sure for some of the sites i'm seeing it in google.nl for, that it is not a duplicate content issue as well.
Some strange things afoot indeed. As a person who is currently promoting two sites, neither of which have previously existed (according to archive.org) and appear on none of the blacklists for google, I am seeing fresh googlecaches every 3-5 days, but still not site: or allinurl:, link: or any other seo specific searches. Google is obviously undergoing some algorithmic changes of late, of what type is to be discerned, but strange nonetheless.
Isn't that the point of the site:uk search? They are just clamping down on it now, I suppose.
Besides, only geeks use that.
I disagree about that geeks part, our tracking shows that as much as 45% of Google UK search uses the UK specific filter. The point of the UK search is to filter out non-UK results, not UK businesses. The unfortunate aspect is I know a number of businesses are being affected by this.
MrCrip, what kind of site are those stats for? I've been wondering for a while what the percentages for the different types of search (international vs. langspecific vs. local) are. It would be good if someone could do some research on that and write it up in a blogpost :)
Rand, that sounds like a perfect SEOmoz post :)
The stats were collated mainly on online stores as we are tracking enhanced data. I will make a post of the findings over on the Fresh Egg Blog when I get a chance.
Cool, thx, i'll be looking around for that one.
Okay, thinking through all the options, and applying what I know from elsewhere, I think this is related to the recurrence of the canonicalization issues that Google had supposedly fixed before.
Both are things that would usually be more apparent on the homepage/root url. I think there's a connection.
I am sure it isnt the cannonicalization issue at play either, because both the sites in question use server side scripts in the .htaccess file to ensure all visits are directed to URLs with www and index.php is always replaced by parent directory name (in all directories).
I still think we should sit and wait. It happened for a bit, got fixed and now is back, and unless G have lost the plot completely this time, it will get fixed again.
It hasn't been fixed. It has been happening since July that we are aware of, and although it never affects all the sites we have been monitoring at once, neither has it ever completely disappeared from all.
Which sites are affected changes, but there's always a few, ever since July when we first took note. As for waiting, that was our first response too, of course. But it has been happening for more than a quarter of the year now, and at this time of year, is a thing that is hurting some of the biggest brands in the UK (as they often favour the .com TLD over a CCTLD)
It's happening to my '.uk.com' address also :o(