Picture the scene...
You wake up, grab a shower, throw on your Moz t-shirt (and other clothes, presumably...), boil the kettle, pour yourself a cup of coffee, switch on the ol' laptop, let your daily rank checks complete and then slowly run through them one by one...
...Yep...
...Yep...
...Ooo, that's nice...
...Uh-huh...
...Yes! Great jump there!...
...Yep...
...Ye- Wait, hold on... What? Lots of red, all across the board? Rankings have either dropped multiple pages or dropped out of the top 100 results entirely?!
Uh-oh. It's gonna be a looong day....
This happened to me recently with one of my clients. Their homepage - their main page as far as rankings were concerned - had mysteriously vanished from Google's index overnight, taking with it a lot of page one rankings, as you can see from the world's saddest and perhaps most unnecessary GIF image below:
This was also the first time that it'd happened to me. Granted, I've consulted on this type of thing before, but usually when it's happened to someone and they approach me asking what's happened afterwards. However, this was the first instance of it where I was discovering it for myself and it was happening under my watch, affecting one of my clients.
This post runs through the steps that I took to resolve the issue. I acted methodically yet swiftly, and in doing so managed to get the homepage back in Google's index (and - with it - its former rankings) in less than 12 hours.
I accept that this is one of those articles where you probably won't even need it until it happens to you. To be honest, I was in that exact situation - I pretty much knew what to do, but I was still a bit like "OMG OMG OMG, whowhatwherewhenwhy?!" in trying to find an article to just double-check that I was doing everything I could be doing and wasn't overlooking anything obvious.
So... Are you ready? Here we go!
Check if it's just that page or all pages on the site
I primarily use Link Assistant's Rank Tracker (with Trusted Proxies) for my rank checking needs, with Moz PRO's rank checking as a backup and second opinion. Rank Tracker allows a 'URL Found' column, which revealed something to me instantly: other pages were still ranking, just not the homepage. Additionally, where a ranking had seen a drop of a few pages (but was still ranking within the top 10 pages/100 results), a different page was ranking instead - in my client's case, it was things like the Services, Testimonials and Contact pages.
This suggested to me that it was just the homepage that was affected - but there was still a way that I could find out to be sure...
Use the 'site:' operator to check if it's still in Google's index
My next step was to use Google's 'site:' operator (see #1 here) on the domain, to see whether the homepage was still in Google's index. It wasn't - but all of the site's other pages were. Phew... Well at least it wasn't site-wide!
Even though I had a feeling that this would be the case based on what Rank Tracker was saying, it was still important to check, just in case the homepage was still ranking but had been devalued for whatever reason.
Now that I knew for sure that the homepage was gone from Google, it was time to start investigating what the actual cause might be...
Check 1) Accidental noindexing via the meta noindex tag
In my experience, this is usually what's responsible when something like this happens... Given that the option to noindex a page is often a tick-box in most CMS systems these days, it's easy enough to do. In fact, one of the times I looked into the issue for someone, this was what was the cause - I just told them to untick the box in WordPress.
In order to check, bring up the page's source code and look for this line (or something similar):
<meta name="robots" content="noindex">
(Hit Ctrl + F and search for "noindex" if it's easier/quicker.)
If you find this code in the source, then chances are that this is responsible. If it's not there, onto the next step...
Check 2) Accidental inclusion in the site's robots.txt file
It seems to be a somewhat common myth that robots.txt can noindex a page - it actually tells search engines not to crawl a page, so it'd only be true if the page had never actually appeared in Google's index in the first place (e.g. if it were a brand new site). Here's more info if you're interested.
To be honest though, given what had happened, I didn't want to assume that this wasn't the cause and therefore I thought it would be best just to check anyway.
But alas... The site's robots.txt file hadn't changed one iota. Onto step 3...
Check 3) Penalty checks
Given that this was my client, I was already familiar with its history, and I was already adamant that a penalty wasn't behind it. But again, I wanted to do my due diligence - and you know what they say when you assume...!
I jumped into Google Webmaster Tools and looked at the recently added Manual Actions tab. Unsurprisingly: "No manual webspam actions found." Good good.
However, let's not rule out algorithmic penalties, which Google doesn't tell you about (and oh lordy, that's caused some confusion). As far as Pandas were concerned, there was no evidence of accidental or deliberate duplicate content either on the site or elsewhere on the Web. As for those dastardly Penguins, given that I'm the first SEO ever to work on the site and I don't build keyword anchor text links for my clients, the site has never seen any keyword anchor text, let alone enough to set off alarm bells.
Following these checks, I was confident that a penalty wasn't responsible.
Check 4) Remove URLs feature in Google Webmaster Tools
Another check while you're in your Webmaster Tools account: go to Google Index > Remove URLs and check that the page hasn't been added as a removal request (whether by accident or on purpose). You never know... It's always best to check.
Nope... "No URL removal requests" in this case.
It was at this point, that I was starting to think: "what the hell else could it be?!"
Check 5) Accidental 404 code
On the day that this happened, I met up with my good friends and fellow SEOs Andrew Isidoro (@Andrew_Isidoro) and Ceri Harris of Willows Finance for a drink and a bite to eat down the pub. I ran this whole story by them along with what I'd done so far, and Andrew suggested something that I hadn't considered: although extremely unlikely, what if the homepage was now showing up as a 404 (Not Found) code instead of a 200 (OK) code? Even if the page is live and performing normally (to the visitor), a 404 code would tell Google that that page "don't live here no more" (to quote the mighty Hendrix) and Google would remove it accordingly.
Again, it was worth checking, so I ran it past SEO Book's HTTP header checker tool. The verdict: 200 code. It was a-OK (pun fully intended - it's a good thing that I'm an SEO and not a comedian...)
Ok, so now what?
Testing the page in Google Webmaster Tools
Now it was time to ask the big boss Googly McSearchengineface directly: what do you make of the page, oh mighty one?
In order to do this, go to Google Webmaster Tools, click on the site in question and select Crawl > Fetch as Google from the side-menu. You should see a screen like this:
Simply put the affected page(s) into it (or leave it blank if it's the homepage) and see what Google makes of them. Of course, if it's "Failed," is there a reason why it's failed? It might also help to give you an idea about what could be wrong...
Asking Google to (re)index the page
Once you have done the above in GWT, you're given this option if Google can successfully fetch the page:
I decided to do just that: ask Google to (re)submit the page to its index.
At this point I was confident that I had done pretty much everything in my power to investigate and subsequently rectify the situation. It was now time to break the news, by which I mean: tell the client...
Inform the client
I thought it best to tell the client after doing all of the above (except for the 404 check, which I actually did later on), even if it was possible that the page might recover almost immediately (which it did in the end, pretty much). Plus I wanted to be seen as proactive, not reactive - I wanted to be the one to tell him, not for him to be the one finding out for himself and asking me about it...
Here's the email that I sent:
Hi [name removed],
I just wanted to bring your attention to something.
I conduct daily ranks checks just to see how your site is performing on Google on a day-to-day basis, and I've noticed that your homepage has disappeared from Google.
Usually this is the result of a) accidental de-indexation or b) a penalty, but I have checked the usual suspects/causes and I see no sign of either of those occurring.
I have checked in your Webmaster Tools account and Google can successfully read/crawl the page, so no problems there. I have taken appropriate steps to ask Google to re-index the page.
I've done all that I can for now, but if we do not see everything back to normal in the next couple of days, I will continue to research the issue further. It's likely the case that it will recover of its own accord very soon. Like I say, I've checked the usual signs/causes of such an issue and it doesn't appear to be the result of any of those.
Just to check, have you or your web designer made any changes to the website in the last couple of days/weeks? If so, could you please let me know what you have done?
I know it's not an ideal situation, but I hope you can appreciate that I've spotted the issue almost immediately and have taken steps to sort out the issue.
If you have any questions about it then please do let me know. In the meantime I will keep a close eye on it and keep you posted with any developments.
(Note: In this instance, my client prefers email contact. You may find that a phone call may be better suited, especially given the severity of the situation - I guess it will be a judgement call depending on the relationship that you have with your client and what they'd prefer, etc.)
He took it well. He hadn't noticed the drop himself, but he appreciated me notifying him, filling him in on the situation and explaining what action I had taken to resolve the issue.
* Recovery! *
Later on the same day in the evening, I did another quick check. To my surprise, the homepage was not only back in Google, but the rankings were pretty much back to where they once were. PHEW!
I say "surprised" not because of my ability to pull it off, but with how quickly it'd happened - I expected that it might've taken a few days maybe, but not a mere few hours. Oh well, mustn't complain...!
The real (possible) cause...
So what did cause the deindexation? Well, another suggestion that came from Andrew while we were down the pub that I'd stupidly overlooked: downtime!
It could've been an unfortunate and unlucky coincidence that Google happened to re-crawl the page exactly when the site had gone down.
I hadn't added the site to my Pingdom account before all of this had happened (something that I have since rectified), so I couldn't know for sure. However, the site went down again a day or so later, which made me wonder if downtime was responsible after all... Even so, I advised the client that if this was a common occurrence that he should maybe consider switching hosting providers to someone more reliable, in order to reduce the chance of this happening all over again...
Preparing yourself for when it happens to you or your clients
In order to make sure that you're fully on top of a situation like this, make sure that you're carrying out daily rank checks and that you're quickly checking those rank checks, even if it's a quick once-over just to make sure that nothing drastic has happened in the last 24 hours. It's clear to say that if I hadn't have done so, I might not have realised what had happened for days and therefore might not have rectified the situation for days, either.
Also, having a 'URL Found' column in addition to 'Ranking Position' in your rank checking tool of choice is an absolute must - that way you can see if it's a particular page that's affected if different pages are now the highest-ranking pages instead.
Anyway, I hope that this case study/guide has been useful, whether you're reading it to brush up ready for when the worst happens, or whether the worst is happening to you right now (in which case I feel for you, my friend - be strong)...!
Also, if you'd do anything differently to what I did or you think that I've missed a pivotal step or check, please let me know in the comments below!
Did you like the comic drawings? If so, check out Age of Revolution, a new comic launched by Huw (@big_huw) & Hannah (@SpannerX23). Check them out on Facebook, Twitter and Ukondisplay.com (where you can pick up a copy of their first issue). Their main site - Cosmic Anvil - is coming soon... I'd like to say a massive thanks to them for providing the drawings for this post, which are simply and absolutely awesome, I'm sure you'll agree!
Hey Steve,
First of all I'd like to congratulate you for this kick-ass post and for the interactive and helpful images. Your tone in the whole article is superb and quite friendly.
Coming to the post, your all points are superb. You're right sometimes we accidentally trigger the whole process which makes Google to remove that page from SERPs. I agree with your friend Andrew here, sometimes due to the maintenance the site may down and unfortunately Google indexed the page at the same time. What do you suggest to tackle this problem?
Lastly, I appreciate the way you informed the client and I think it is the best exercise that each one of us should adopt.
Thanks Umar :-)
I suppose the best way to handle the risk of Google removing your page(s) while your site is down is simply to reduce the chance of your site going down. This may be determined by the reliability of your hosting provider - if your site goes down all the time, maybe enquire with them, and if it continues, maybe consider switching hosting providers to someone more reliable.
When it comes to maintenance... I'm not an IT guy, so I don't know if - with some changes - sites have to go down no matter what, but when it comes to things like making changes, site migrations, hosting provider migrations, etc., try and do the process as seamlessly as possible in order to reduce the chance of downtime. Hope that all helps :-)
Hey Steve, Thank you for your valuable response. I think if someone wants to play in Digital Arena they should make sure that site is available round the clock.
I wish to see your this post hitting the main blog soon :)
Hey Umar, I would like to recommend you that if your site comes down to maintenance then you should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.
Good advice - thanks for jumping in, Rameez :-)
My recent project had this issue. The server was returning 503 not available error for a while on several pages and some certain pages were working normally. Eventually, I lost rankings for all those pages that were playing hide-n-seek with Googlebot and users.
Gags around your post and the detailed description of your problem was impressive. At present i am Handling a website with 2000 webpages and for around 750 pages i am having great PR. But my PR has been playing with only 3 Numbers 3,4,5 it revolves in this 3 ranks, And the similar problem i am facing when every morning i came and check the PR not s single morning goes without surprise. People now a days introduce lots of penalty terms like "Sand box" "Google Dance" etc, But Our client doesn't understand the same ( we all are sailing in same as we know how they are realized actuality of SEO) .
As you said i prefer Four Simple steps when it happens to me:
1) First is URL : check as you mentioned
2) Second is Google webmaster tools and check the over all impression and and crawling defects for the pages.
4) Check On-page terms each and every, follow the same with security check.
3) fetch as Google and submit to index. If this will be success full then sit and relax you will be back on PR. IF not Serious things have happened be patient don't panic believe on Google and website. Go for competitors check.
Great tips there - thanks for sharing, Deep :-)
Hi Deep,
As suggested, we have done all the procedure for our website but still its ranking going down. Also, checked competitors Back-link we have more quality back link then competitors. Seriosly, we have the same back-links where our competitors have, but still we are fallen down.
Hi Steve,
great post!
I'll give my 50 cents on the subject-
I was surprised I couldn't find any mentions of the canonical tag in the post nor in the comments.
Actually, I have a long experience with pages suddenly being removed from Google's index and I can tell you that from my experience the canonical tag is the most common problem.
The main thing with the canonical tag it's like a scalpel knife - it's very powerful, but can easily harm your site if it doesn't being handled properly.
I think the canonical tag deserves it's own chapter but the most common mistake I've noticed throughout the years is implmenting the canonical tag the direct to wrong page (msotly the homepage).
Actually, I had a case when all the canonical tags (sitewide) were directing to the homepage. The outcome? only one page left indexed - the homepage. Fortunately, it was a small site and I was able to monitor the problem soon enough.
Wow, that's a brilliant point, Roey - thanks for mentioning it! It just goes to show that there's no 1 or 2 things that could cause such a problem but more like 5-10, haha! :-)
Are you talking about when rel="canonical" is implemented incorrectly, e.g. you want pages A and B to rank, but page A's canonical tag points to page B, and therefore Google removes page A from its index? I've never had first-hand experience with such a problem but - like you say - I bet it happens often and is a common culprit. If you have any examples of links of case studies where such issues have happened then please leave another comment passing them on. Thanks again!
yes, to your question - that's the case.
I'll give you another example -
I have a client who just implemented a new forum on it's site. I asked that every thred will have a self referral canonical. After a few days I started to to notice a huge droppings in the site's rankings. After I check those pages I quickly found out that the canonical tag to the main cateogry pages (not the forum) were directing to the main forum page. And just like that - they were deindexed.
Haha, yeah, I've heard of that - people who think that you can canonicalise folders by simply pointing to the root folder or the root page (the homepage). How did they do when it came to recovery, after the problem was fixed? Did it recovery quickly or take time?
It was very quick actually but it highly depends on the size of the site - if you have over million deindexed pages - you're in trouble mate!
Hello Steve,
Thanks for sharing such an informative post. It will be of great help for every one. Just wanted to add one more point which you might find worthy to keep under your check list.
A year back something similar happened with me for one of my Irish clients. A page ranking on top for a very critical keyword disappeared overnight. However we were not as lucky as you are as it took us a week to get it back on SERP, exactly on the same position. I ran all the checks you mentioned above above however once I recovered I felt the reason for de-indexing in my case was different therefore I advice for then onward to every one to perform this check as well.
In my case we had two different pages but now I can say they were similar in some way if we check the content were being targeted. Page X was on top position and Page Y was on the second page of the Serp. Once page X disappeared we ran an extensive check and to my surprise on the very second day I felt may be the content is the culprit. And guess what.............. the moment the content was changed we got back our page. A year back I would acknowledge that Google was quite smart with Panda but the thing which happened to us was something un- expected. But it happened so next time if your page is de-indexed make sure that you have all pages of your site unique a 25% similarity can even lead you to something unexpected.
Hi nomadpandey, thanks for sharing your story. That's a really good tip as well - if it is Panda-related, make the effort to make sure that the pages are truly unique (not just slightly unique), as it could make all the difference.
Haven't read all the comments, so apologies if this has been noted! :-)
For #1, don't just look for "noindex". The robots meta tag could be set to "none", too -- which is the same as a noindex. <meta name="robots" content="none">
And don't stop there. Look at the returned headers, not just the source code. Do a Fetch as Google and then click on "Success". You'll see what googelbot sees. Check to see if an x-robots-tag is set "noindex."
Wow, great suggestons - thanks Suzany!
Steve,
Thanks for laying out your process. I have found that if a site dissappears without cause from a manual action or algorithimic hit it typically does not take long to return to rank well. This can be of comfort to a client worried about loss of business.
Hi Jared, thanks for the comment. That can be true in many cases, and often it can be a case of worrying for nothing, but sometimes it's worth doing a few checks and doing your due diligence just in case it's something more severe... :-)
Hey Steve very informative post! A customers page disappeared from Google's index this weekend so I followed the directions you gave and it said that the url was removed so I re-submitted it... Everything seems fine its back up yet is there an uncorrected error to be fixed or they just do that to Cause panic? Is there a logic explanation to this?
Thanks
Hi Trevor, thanks for the comment - glad to hear that it's back up in your case.
It's really tough to say... the words "Google" and "logic" don't tend to go well in the same sentence! ;-) Seriously though, there could be any number of reasons why these things happen, I mean I'm not even sure if downtime was responsible in my case - I can only surmise. I can't really say much more without knowing more about your customer's particular circumstances, but then again, if it's recovered, maybe it's not worth investing hours and hours finding out the reason if it's unlikely to happen again.
Did you check your analytics? If the outage was long-enough, it might just show up there. The higher the traffic to the site, the more obvious it'll be. Failing that, can you get your hands on the server logs?
Hi Doug. Great suggestions!
I just checked the client's GA account on the dreaded date and there doesn't seem to be any higher or lower traffic than usual.
I'm not sure if I have access to the server logs, but it's something that I'll certainly look into and also keep in mind for future clients. Thanks! :-)
Using the Header Checker, as you have suggested, I am seeing a "302 moved temporarily" for the homepage. Any idea what that is?
Hi BBuck - the homepage for what, sorry? Which site's homepage?
OH! Sorry, I forgot: www.fairfieldfamilylaw.com . it has performed strong for last few months, and suddenly has dropped off a cliff. I think it has to do with bounce rates due to reachlocal campaigns my client agreed to without me knowing. But I want to look at every possibility.
Oh I see what you mean!
Yeah, if you check it on the tool's default User-Agent (MSIE 7.0), it 302 redirects to this URL, which says that Internet Explorer 7 is unsupported. If you try one of the other User-Agent options, it should be fine and reporting a 200/OK code (I tried Firefox 1.5). It could be the case that Google is also seeing the "?tmpl=unsupported" link and therefore that could be responsible for the drop? I'm not 100% sure though.
The main question: is it really "not supported by [the] website?" If there's a way to make the site 'supportable' (if that's a word!) with IE7 then maybe look into doing so, or perhaps it an unnecessary error message in the first place (e.g. it's been placed by the website developer but the site is actually fine in IE7)? Anyway, I hope that helps. If not, you might want to try suggesting it in Moz's Q&A section (if you're a PRO member) :-)
It's a rocketheme joomla 2.5 template, which has discontinued support for IE7.
Ahh I see... I don't know much about Joomla unfortunately (I'm a WordPress man myself) ;-)
Is there the scope/flexibility to change the template to something supportable? Or perhaps is there a way to bypass that message (and take the risk with showing the site in IE7)? Not sure what else to suggest otherwise!
Thanks for your help Steve. I'm digging in to see what I can find.
Happy to help :-) Good luck with it! If you happen to get it sorted, please let me know!
Hi Steve,
I'm exactly going through same situation with one of my client website but in my case page is index in Google, No manual penalty(Checked Webmaster) no algorithm penalty(Checked backlinks), Fetched page in Webmaster no error found previously page is ranking on 11th position but now replaced with some other page of website and ranking on 7th page of Google and this all happened after 28Nov.
Hi Sachin. Hmm... If the page is still indexed then there might be some other reason for it. Are you sure that there's no algorithmic penalty associated with Panda (i.e. duplicate content)? Otherwise I'm not sure what else to suggest off the top of my head, especially if you've double-checked the above steps, just in case. Maybe keep an eye on Moz's Google Algo Change History just in case Dr Pete adds in something around the 28th, or alternatively try Moz Q&A. Good luck! :-)
It was ok before 28 Nov but went down after, I visited Mozcast and Algoroo and could see flux on 27th Nov in Algoroo.
I see... Maybe keep an eye on Moz's Algo History, just in case they add something around that date (and can make a guess as to how sites were affected). Search Engine Land often reports on algorithm changes, too. All the best with it!
Ok, Thanks Steve.
Great post Steve. I have a comment and a question. This scenario has happened to me. I was working as an in-house SEO at the time. The CEO gave me a directive to add some very spammy (IMHO) keyword anchor text links to the top of the homepage to try to get better rankings for those terms. I voiced my reluctance to do so, but she was adamant so I added them. At the same time, I thought "What positive impact could I make on the SEO of this page that no one else really cares about?" - The answer was title and description. They were awful, full of "stop" words, too long, poorly written. To me they were the low-hanging fruit, so on the same day as adding these spammy links a rewrote the title tag and description of the site's homepage.
The next thing I know, the very next morning the CEO is standing at my desk asking me why the homepage is no longer in Google. Yeah, I about peed my pants. She asked me if there was anything else that had changed, and I honestly answered that I had re-written the homepage's meta tags. Oh boy, she was NOT happy. Up until then, no one had ever breathed a word about meta tags. I was not only told to immediately change them back to the awful old ones....but she actually brought over another member of the marketing department and told her to stand there and watch me do it. Sooooo humiliating.
Within 2 hours the site was back where it had been. I didn't do any of the steps you outlined above.
So here is my question:
Is it possible that a web page might disappear out of Google's index for brief periods occasionally if something on the page has changed enough to warrant a new cached version of the page?
Has anyone else experienced this or anything similar to the scenario I've described above (sans humiliation I hope)?
Blimey, that sounds like it was a scary experience!
I honestly couldn't tell you. I wouldn't have thought so, but based on what you're saying, it could very well be the case. I've just not heard anything like that happen before, simply because you made a few meta data changes.
I wonder if the meta tags had nothing to do with it and it went down (and recovered) of its own accord anyway, but my changing the meta tags back, your CEO et al truly believed that your change was responsible? Something to ponder... It's interesting! :-)
Thanks Steve. Yes, the same thing crossed my mind.
Something else you mentioned was site downtime. This brings up a separate question that I meant to include in my comment.
I am intrigued by the idea that you believe it's possible the homepage you were dealing with disappeared from the SERPs due to the site being down. This is why I ask:
Every day between 2:00AM and 3:00AM our site completely reloads itself. While it's doing this, the site is "down" for 10-15 minutes...every single day. I have irrefutable evidence (via server logs and Google Webmaster tools) that googlebot occasionally is crawling our site when it goes down.
Could this result in pages that googlebot can't access during this time being removed from the SERPs?
What other negative ramifications could this be having on our SEO?
Thanks again for the excellent post!
Hi Dana. While I'll admit that I don't know the specifics on how Google handles different error codes (e.g. if it sees a 404, does it take it down or assume it's just a temporary outage?) but I guess it makes sense that if a page isn't available when Google visits it, it's going to think "huh, do I really want searchers landing on this?" I don't know for sure though... Like I say, I can only speculate that the drop was due to downtime (but I'm not even sure if that's the case)! :-)
Nice post Steve (really like the illustrations) - always amazes me how many times I've seen developers upload a wrong robots.txt file from a test server to a primary one, always the first place I check!
Thanks Matt! :-) Absolutely, the same goes for accidental noindexing - it's easily done, so it's worth checking first thing.
Its amazing how you tackled the situation, if it was for me I would definitely panic about that fact that the most important page was just gone!!!! but knowing that you can get it back with how you reacted proactively :) I just know even we are able to do so. But fortunately I have not come across such situations but If I do at least I know what steps I can take to get it rectified or at least make my self satisfied by doing these checkups that some things such as getting hit by one of the penalty checks is not a reason for the situation :) . Thank you for the article.
One of the most important points here is, to check Remove URLS in Google Webmasters Tools as it is little in-depth and people don't check it.
A client contact me to check why one of his services page doesn't appear and all the other webpages are doing fine. I was quite surprised that time thinking I got a challenge lol. But when I studied the website, that page was removed from Google Webmasters.
I just sent re-included it and the next day I it was indexed and ranking for his targeted keywords. :)
Hi Kumail, thanks for the comment. That's a very good point - and in fact it nearly didn't make this list originally (it was a last-minute addition). So it's good to hear of an example where that was indeed the case and what was responsible for the de-indexation. Good stuff! :-)
Thank you. I don't know exactly why or who did that but it was really a good find - after digging the complete source code, links, robots, filezilla :)
Yeah... Perhaps they didn't see "Remove" and thought it was a list of high priority URLs - or they thought it said "Do Not Remove" :-)
hahahaha! For sure :D
I had a similar problem back in the day but it had to do with something entirely different.
My start-up is called MarsBased (www.marsbased.com) and was developing a now-discontinued project called Woliva. For technical reasons, the Woliva project website went live before the MarsBased one, and was properly indexed by Google. Half a year later (or even more) we launched the MarsBased website and decided to kill Woliva. Thing is, we didn't set the redirections right at http level. We just told the hosting to redirect all Woliva.com petitions to Marsbased.com. Google decided that he had already indexed Woliva.com and was not indexing MarsBased.com because they both were the same page, to avoid content duplication. Therefore, all MarsBased was indexed except for its homepage.
It was very hard to detect, as Google Webmaster Tools would offer no details nor errors. When asked to index the homepage again, either via sitemap.xml or via fetch as Google, it would always return OK (as in 'no errors') but never index it. Needless to say, the page returned a 200-OK.
Web redirections might be something worth checking, after all.
Hi Àlex, thanks for sharing your story with us. That's a good point actually - I imagine that improper redirection can cause all sorts of trouble. And yeah, if Google considered it a duplicate, then Woilva would've been seen as the original an MarsBased the imitator. I take it you got it sorted in the end? How did Google treat the homepage once it was all fixed? Did it rank ok?
As soon as we took Woliva down, MarsBased was indexed within hours. Now it's everything fine as it should be :)
Ahh I see, that's cool. Thanks for the follow-up :-)
Steve Morgan, You have done a great job and it is the best practice to handle a page hit by Google, also your tone and the choice of pictures are superb attractive and too interactive also your decision to tell the whole story to the client shows your honesty with your client and especially with your work.
Thanks Spook :-)
I am suffering this issue on just one individual landing page, and have already tried everything on this list. Has not been indexed for 3 weeks or so now and am running out of ideas!
Thanks Steve, great post and nice illustrations!
Thanks Ulyana - yeah, I love the illustrations, I'm so glad I got them done! :-)
Thank you for the post. Very well written. I hope I won't be needing it though, as I suspect that my clients wouldn't be dealing with me that leniently...
Haha, well I think I got lucky as it was this particular client who's very patient and understanding. If it'd happened with another client of mine, I think it would've reflected the first pic in that section more accurately... :-)
Great article, Steve! I have a (very) small business and can't afford an SEO expert, so I have been learning a bit on my own. Almost afraid to even post this since I'm such a newbie. :-) Anyway, I found the article very informative and learned about even more topics I need to learn more about!
Hi SDD! Don't worry about that - the Moz community is very welcoming in my experience and I was in the exact same position a couple of years ago (we all have to start somewhere)! :-) I'm glad the article helped - fingers-crossed you won't end up in that situation anyway, but if you do for whatever reason, hopefully it will help.
Hi Steve! I will definitly bookmark this post of yours. Luckily, I have not experienced it but if ever this will happen to me, I now know what to do. Glad that I have found this very informative and helpful post of yours. I will also share this to others. Please keep us updated!
Great Stuff Steve but i want to know how much time your website took to come back in last position? Generally the ranking suddenly drops but their recover take many months.. Was it happened in your case?
Hi Sweta. It was within about 12 hours - I couldn't believe how fast it was! I checked later that evening just to double-check something and saw that it was back. I too was expecting it to take days, week, or even months, so I was very fortunate that it was so much faster.
Thanks for sharing your experience Steve.It also happened to me few months ago but I'm glad that I resolved the problem after a couple of hours.
I have next to no science to back this up but I ensure the site's corresponding Google+ badge is site wide. Feels like a way of keeping a site in Google in case it suddenly isn't in Google when a client accidentally disallows Google in a Robots.txt file....
That makes little sense, I've been at the mulled wine ;-)
Hi Graeme, there's theories that being on G+ and sharing content on G+ can help with (faster) indexation, so I guess you never know... ;-)
Talking about downtime, your site is down: downforeveryoneorjustme.com/seono.co.uk
Hah! :-) Seems fine now - thanks for the heads-up. Let's hope Google didn't notice eh? ;-)
Did you check Bing at all? Even though it's not a major traffic driver, they also have a good webmaster tools section that could give you additional clues, and it could have helped diagnose if this was something that happened across all engines or just Google.
Very glad that this was resolved for you so quickly, and I really enjoyed the post.
That's a very good point! No, they're not on Bing Webmaster Tools, but that's something I'll have to rectify going forward. Thanks! :-)
Awesome post. I've had several pages disappear from Google and it usually starts a panic attack and no real plan of action.
Hehe, that was kinda my strategy secretly... ;-)
Thanks for the walkthrough! Definitely a great help when that day comes along. I never knew Google could accidentally de-index your site if it crawled your site while it was on downtime. Surely Google must've recognized such a mistake later? I guess not, but good to know!
Hi James. Truth be told, I'm not 100% sure - it might depend on the circumstance. Maybe I (and my client) got unlucky. I'm sure someone else can jump in here and leave a comment giving information on Google's handling of downtime though! :-)
Well, even if that hypothetical scenario did occur, wouldn't Google's algorithms pick up the activity on your site again once it was back online? I thought Google operated in real time. It's definitely something to wonder!
Right Steve. Website up-time play a very crucial role for continue indexing in Google. As far as website downtime increases day by day, Google may pass very less value(importance) to your website as it did. Very good point! And probably, it may affect your ranking too.
Congrates for recovery, and thanks for this perfect post.
Thanks Gaurav :-)
One tool I've used to make me aware of changes to the code (usually monitoring robots.txt) is Pole Position's Code Monitor at https://polepositionweb.com/roi/codemonitor/index.php. It's free, and checks the code of a website once a day for changes. It's quite useful if you're not always best buddies with everyone who may have access to a site. It's all done on their end, so there's not a need to install anything on the site. I have no relation to them, it's just a tool that's saved me grief several times.
Ooo that's cool - thanks for sharing, Keri. I'll have to check it out :-)
I had the exact same thing happen to a client one day. Let me tell you that is not a fun phone call to get! When we started working with them we knew they had a partial manual action penalty but to see just the homepage vanish with nothing else changing was frustrating and frightening, to say the least. No other pages appeared to have been hit, and our best guess what that the homepage had been singled out because their previous SEO firm had built a lot of links with keywords as anchor text specially to that page.
Yikes, so it was a homepage-aimed Penguin penalty? Sounds nasty...! :-/ Hopefully the client understood that it due to their previous SEO firm's efforts and not yours though!
Great step-by-step process Steve and bookmarked
Love the creative story-telling visuals, especially the angry client face we tend to picture when things don't always go to plan :)
Client communication is key and having a process enables us to check things, carry out audits and keep a focused, clear head without emotion getting a grip.
Thanks for sharing!
Thanks Tony :-) I'll be the first person to admit that I'm a massive worrywart(!) - I was dreading telling him, so for me to discover that his response wasn't to throw things at me was a big relief. I think we get it in our heads that no client likes bad news, and while that may be true, things can't go perfectly 100% of the time, so it's all about how you approach it and how to tell the client in those difficult circumstances.
Great article Steve, nice graphics (they must have taken some time!)!
I enjoyed the thorough and logical approach to establishing why your clients traffic had dropped. It's super annoying that there is not always an apparent reason as to why your clients site may have gone down. From your steps, I'd struggle to see another way in which your clients site would have gone down other than the downtime as you have suggested.
Thanks Sam - the graphics are courtesy of Age of Revolution (there's info about them at the end of the post) :-)
That's true, it is frustrating that I can only make an assumption as to the cause of the page's disappearance, and don't have anything concrete to go by. But such is the life of us SEOs - we live by assumptions more than absolutes, hehe!
Haha, that is true... the majority of SEO work is based on assumptions and that's not likely to change anytime soon (especially with 100% not provided), although I think you have given a good outline of all the things we have to do before having to resort to assumptions in your article!
Thats what we do when a ranking is away - or when a site is out of index. But there you only guess what the reason was - everything was ok. If google would have crawled the site just when it was down, you would have seen crawling errors in google webmaster tools i guess.
Hi PND, thanks for comment. Do you have more info on this please? I think it could be useful to share here :-)
This is a pretty good case study, good work Steve. It's that horrible "BUT I DID NOTHING WRONG!" moment, isn't it!
Also way better advice than the old "Not indexed? Share it on Google+ then!" solution.
Haha, you've guessed it, Iain! As SEOs we take responsibility for this sort of thing, even if we weren't the ones responsible for it. I'm just glad thatin this case my client took it so well, but there might clients out there who may not do so, even if it's not the SEO's fault, and I feel for those guys...
Interesting case study, thanks for sharing Steve. I can imagine not all clients would take that news as calmly as yours did. Glad you were able to get the page re-indexed quickly.
Thanks Shelby :-) Yes, you're right there - I think I'm very lucky to have a such an understanding client. I can imagine some clients being less understanding/forgiving, especially in more popular and/or competitive industries...!
Great post, Steve! I had a similar experience with a client where the client was poking around in GWT and noticed that they had 0 pages currently indexed by Google. He sent me a screenshot and being that the client's site was fairly large - roughly 1,000+ pages - this didn't make any sense. I did a quick "site:" check in Google and showed the client that most, if not all of their pages were still indexed and hadn't lost any rankings. I still don't know what caused the bug in GWT, but within a month, the indexation stats returned to normal.
Thanks Brandon! Wow, that's a fear-inducing bug if I've ever heard one! Sounds like you kept your head though and thought to check if they were actually affected or not... Fair play, the 'site:' command comes in handy eh! :-)
another superb technique added to my storage (my mind)
Best place to store it, Mark! ;-)
Great visuals! I'm sure any SEO can relate with having a KW drop off Google. My first step is to always check if there's an error with the analytics code. Then it's time to play detective.
Ooo that's a good suggestion, and one I hadn't considered previously. Thanks Takeshi! :-)
I had the exact same thing happen to a client last week, a site home page was de indexed. I pretty much did what you did step for step, and a couple of hours after re submitting to Google it was back in the index and ranking again. Quite odd I thought and the only think I came up with was also down time.
Great post.
Thanks Maximilian. It's good to see that the same steps worked well for someone else as well, and that you also managed to get it back in the index pretty sharpish! :-)
Super helpful, Steve. Though in regards to downtime: isn't that visible in Webmasters as well (under "Crawl Errors" > "DNS")? We've had even brief amounts of downtime reported to us by Webmasters via email, so it seems odd that you wouldn't have that there.
Checking the Content Keywords can also be worth your time — we've had a couple of cases where no manual webspam actions were detected by Webmasters, but our #1 content keyword, was, say, "cialis", due to a code exploitation on the site, and that definitely has some effect on Google even if it isn't outright marked as a threat. (We've found this tends to happen a lot with sites that have Facebook widget embeds in particular.)
Thanks for the comment. I've just checked my client's Webmaster Tools account under Crawl Errors > DNS and on the date that this all happened, it's reporting 0 errors - so are you sure that that's absolutely the case?
Nice one Steve. Loving the cartoon versions of Ceri and I.
Thanks for the shoutout too! :)
Thank you sir! In fact, thank you for the suggestions of the possible 404 code and downtime. I'm a little embarrassed that I didn't think of them, but at the time, I was kind of in panic mode, and I guess we never think straight what that happens :-)
Aha,I like the Moz T-shirt..............
Hehe! I managed to get one at BrightonSEO earlier this year, although my wife's since stolen it and adopted it as a pyjama top. Typical eh?
They are also great to go under scratchy sweaters in the winter, too!
Lol! Well I guess it was put to good use :)
Hi Steve,
Good little check list here! This hasn't happened to me yet (touch wood) but if/when it does I have bookmarked this to go thorough these checks.
Great article really love the style!
Good stuff, thanks Jonathan! :-)
Why did I have to jinx myself! 2 Hours later clients site got hacked :-|
Oh no, you're kidding! I hope you managed to get it sorted without too much grief! Good luck!
Fantastic article great job!!
Simply superb!
Thanks For Sharing
Registered tjust to say THANK YOU!!!!. Checked to see if i was still number one in google to find my homepage didn't exsist in google, came across this site and got to the last check which fixed it immediately.
I tend to panic when this happens...but in my experience the site typically jumps back and ranking higher than prior to the disappearance. Â
Don't go straight to the conclusion that the page has been penalized.
Checklist 1 : Check if the page is still running or not.
Checklist 2 : Check the index and cache status of the page.
Checklist 3 : Check the url parameters.There is a possibility that because of the url parameter the page is no longer indexed.
Checklist 4 : Check the robots.txt file.
Checklist 5 : Check whether noindex has been mistakenly mentioned on the page.
Checklist 6 : Check whether rel canonical to another url has been mentioned on the page or not
Hello steve,
please help me because the last 2 days i will get crazy.
I am managing the seo of an eshop, www.1sexshop.gr [NSFW] and the last 2 days i noticed a big change in serps for some keywords. The seo is going in google.gr for 5 keywords and the eshop is based on magento 1.7.Before 5 days, i set up ssl certificate in the eshop in order to get better ranking results.Â
I setup the ssl and followed the instructions from the magento to make permantetly 301 redirects from the http urls to the https. When i saw in serps that the rankings for some keywords going up-down, i checked some urls to see if all going well with 301 redirects and found that the redirects where 302 and not 301, its a big bug from magento.
I solved the problem from the htaccess but still the rankings for the home page have dissapeared. I fetched again site from webmaster tools an i am waiting to see.
Can you please help me and advise me what else to do?
The rankings will come back or i have failed.
Thanks
I came on this article because I have the trouble that my homepage disappeared from Google since +/- 2 weeks. I checked all your steps, but unfortunately, nothing seems wrong. I'm not that lucky that it's back in one day, it's already 2 weeks now.
Goo info thanks - this has happened to only my home page showing under one key phrase it has gone! - Any ideas?
Nice Post Steve; One of my friend has a same situation. He shift his whole website to asp from html and put 301 redirect on whole website. the new homepage was example.com/home.asp and when I check the example.com url it shows 200 status code instead of 301 redirect to example.com/home.asp. I ask him to immediately fix it and submit it to Google index again. But he haven't seen any positive result till now.
Nice Post & Same thing happen to me also. pretty much did what you did step for step, and a couple of hours after re submitting to Google it was back in the index and ranking again....
These days i saw this many time, great tips to get back to life :)
Congratulations you did a great job Steve.
Awesome Post..It's super annoying that there is not always an apparent reason as to why your clients site may have gone down.
Great, we all need this stuff to prevent our important work to disappear. Like the steps you posted. :)
Good job but how I'm looking for algorythmic penalty:
- checking serps - keyphrase/s dropped out of top100 or at least -40/50
- checking # of visitors from google organic - if dropped significally in 2-3 days (at least -30%)
- checking links to website if they're still exist and site:domain.com isn't 0
- and looking into robots.txt, htaccess, dead links, potential dc on my website, html errors in gwt, sitemap.xml errors
Ahh well an algo penalty's a whole other kettle of fish! :-) This probably isn't the article for you sadly - this is more for when it's not penalty-related (and you know that to be the case for sure)...
What do you think might be the cause - duplicate content, heavy keyword anchor text usage, or something else?
i am having similar issues. my host went down for 24 hours and 5 of my top ten ranked websites lost their homepage rankings. Article pages retained same ranking. i am puzzled and don't have a clue what to do at the moment.They all are high quality websites if any of you have similar experience please advice.
Hi amigo123, have you followed the steps outlined in this post? It could be the case that that particular page is no longer indexed and you can resubmit it using Webmaster Tools. Good luck!
yea, i did fetch with google and submitted URLs to index. Unfortunately it is still same. Nothing on manual action so it must be some kind of penalty. All hard SEO works goes in vain. What can i do. i am just waiting LoL.
Awesome post