"I know before the cards are even turned over..." - Mike McDermott, Rounders
When Mike McD was called by Teddy KGB in a huge No-Limit Hold'em poker pot, he didn't have to see his opponents hand to know that KGB had two aces, the only hand in the deck that could beat his nines full of aces (if you have seen Rounders, feel free to skip over the video below, if not, you probably should get on that). This was the same feeling I had when we got "SERP a DERPd" via accidental noindexation of 9,000 of our most important pages....
Contents:
- What happens when pages are accidentally noindexed
- Tactics for getting pages into the Google index quickly
- How noindex impacts SERP rankings
(note that I am focusing on Google in this post)
Background:
I am an in-house SEO and customer acquisition marketer at SeatGeek.com, a NYC tech startup. Our site is a ticket search engine for sports concerts and theater tickets (i.e. "a Kayak for event tickets").
On Monday 8/1, I was searching Google for 'mets tickets' and saw that SeatGeek had slipped from page 1. Worse, we weren't even on page 2. I tried a few more queries that I knew we should be on page 1 for and still nothing. My heart was beating. Had we been Panda'd? It didn't make sense, but I was panicked. Then it hit me. I opened up our New York Mets page, but, just like Mike Mcd, I knew before I even clicked view source...content="noindex" on all of our product pages.
I have only been doing SEO for ~2 years, so I had never directly experienced an accidental noindex situation. So even as I read reports of these not having an impact on rankings and knew this wasn't as bad as an accidental canonicalization problem, I couldn't help but envision the worst case scenario...9,000 of our most important conversion driving pages would be out of the index for weeks and would not have their same rank when they got back in.
What happens when pages are accidentally noindexed
This is a chart of incoming organic traffic to one of our key pages right when the noindex hit.
Obviously organic traffic ceased to exist. Interestingly though, Google Analytics still reported some traffic to these pages.
This might be the one instance where having less frequent crawl frequency can be beneficial (assuming bandwidth isn't an issue). The pages that got noindexed are recrawled every 4-6 days, which would have given us a buffer if we caught this sooner. Unfortunately, Google waited until Saturday to crawl these pages and we didn't catch the problem until Monday.
Reindexation Plan and Tactics:
The first course of action was to remove the noindex tags, which one of our pop star engineers did within five minutes. This was right around the time I sent out my first plan of action email which I have included below in case you ever have to write the same email:
All,
So I was doing a daily scan of SERP positions and started noticing team band pages had dropped. At first I thought we got Panda'd, but it looks like the noindex tags that are supposed to be applied to search pages and filtered navigation recently got pushed into production, but because those pages only get reindexed every 3-6 days there was some delay in the traffic impact, which you can see if you filter by team/band pages.
We are currently:
- Noindex already removed in production
- Writing blog posts that link to all major sports teams to get these reindexed (more difficult for bands)
- Launching social media campaigns to support this cause
- Forcing update on .xml sitemap (hopefully to help with concerts issue)
- Investigating additional techniques
- Going to look into the current traffic impact / which pages got impacted the most (hopefully some deeper artist type pages never got recrawled before the fix)
https://www.webmasterworld.com/webmaster/3601620.htm Here's to hoping this is true "My experience is that "noindex" is quite harmless when it comes to ranking. As soon as you change it to "index", the pages should pop up at nearly the same positions in the SERPs as where they were." I will keep you all posted. -Chad
Even if rankings would come back, we wanted this to happen as quickly as possible. I had a plan, and fortunately some great interns to help me out. So this is what we did (excuse any repetition from the email)...
- Removed noindex tags
- We ran social giveaways and campaigns generating strategic social shares
- Internally, we set up a Google doc with our most important product pages as rows, and everyone's names as columns and had people +1 and Tweet various pages from the list, checking off as we worked through it (read more from Rand about the power of social shares for SEO)
- Wrote / published blog content that linked off to all the pages that had been removed from the index
- Examples of the previous bulletpoint: NFL 2011-12 Season: Average Ticket Prices, Weekly MLB Price Rankings, A Sneak Peek at NBA Ticket Prices, First Look at NHL Prices and 3473 Artists Ranked By SeatGeek's Popularity Score (yes I know Google doesn't go beyond ~+100 links per page...)
- Used our relationships with NY Daily News and Business Insider to get some content up linking back to our most important pages
- Resubmitted our XML sitemap index file via Google Webmaster Tools
- Manually submitted select pages via Google Webmaster tools 'Submit to index' feature
All of the above was completed within one hour of us discovering the issue, except for the guest posts and contest which were done over the next 1-2 days. And then we waited...
Reindexation Metrics:
It took 1-2 days for our most important pages to get back into the index, which we were really happy with. Some of our deeper / less important pages took up to 5 days to come back or longer in some cases. Fortunately we had followed advice from other Mozzers and introduced multiple XML sitemaps earlier in the year with all our product pages in one XML sitemap we were able to easily track indexation of these pages via Google Webmaster Tools. Indexation and traffic were on their way back up by the next day, but as you can tell from the graph below traffic didn't return to previous levels to about 2-3 days from when the noindex tag was removed.
Rankings Impact of Noindexation:
Now let's look at how this impacted our SERP rankings. The example above, was a truly interesting case because our Mets page returned to the index the night of the fix and I emailed my bosses to check it out as a good example of a recovering page, but by the time we got into work the next morning it had left the index again and I looked like a clown shoe. Fortunately, the page came back (again...) into the index the next day and was back up to its previous ranking by the end of the week. This is an example of a trend I noticed that many pages would come back into the index first and then return to ranking for their target terms a day or so later.
The example below is one where we returned to the index but without the same rank as we had before. There isn't really a way to tell if this was impacted at all by the noindex situation, I suspect it was just a random Google dance related to the more frequent shakeups I have seen in event "tickets" related queries. Overall, our page 1 SERP positions have completely returned to prior levels.
Conclusions:
- If you accidentally noindex pages on your site, of course they will stop getting traffic from organic search, but this will be dependent on the crawl rate of the pages (in our case it took ~5 days for them to drop out of the index) and 2-3 days for them to return to normal levels
- If you have a blog that gets crawled quickly, use that as a tool to help drive spiders back to the pages that were noindexed with strategic internal linking (of course wait until you have removed the noindex tag)
- Take advantage of friends & family to help with social shares and pump this up with a social giveaway
- Use Google Webmaster tools: 1) XML sitemap resubmit 2. Manual 'Submit to Index' 3. Sitemap indexation tracking
- You should have Multiple XML sitemaps set up into logical buckets for indexation tracking to faciliate the indexation tracking mentioned above
- Although your rankings might see short-term "dancing", an accidental noindex will not have a negative impact on them
- Lastly, don't be too worried, just follow some of the tactics above and you should be back in the index with the same rankings (have your boss email me if they are giving you crap - [email protected])
Ok so that was probably too much information for just an accidental noindex situation, but when it happened to me it was scary and there wasn't a solid documentation on what to expect, so I wanted to produce this for the next person in my situation. Thanks for reading. Connect with me on Twitter if you are so inclined.
Funny story: A client of mine fired a web developer from managing his ooh-too-complicated ecommerce site, and soon after we started to see a major drop off in indexed pages and organic traffic. Because the drop off was relatively gradual, we spent a lot of time chasing possible explanations.
However, after a few weeks of thinking and hunting, we finally found the problem: The developer we fired (or someone with an awkward sense of humor) had added a noindex tag to a special version of the header template. This version of the header was only served to bots, something that wasn't immediately obvious to me or anyone else I asked to look the site over.
It turns out that this ecommerce system (which shall remain nameless) serves a different version of the site to bots in order to keep from assigning the bot a session id...the reason being that session id's would screwed up indexation because the IDs were in the URL.
Anyways, the point here is that you should ALWAYS FETCH YOUR SITE AS GOOGLEBOT WHENEVER YOU HAVE A PROBLEM. You never know what you might find! :-)
That's also a great advice Jason. I'm currently running into this kind of problem with a "given" e-commerce platform we have... ... (Running to Webmaster Tools)
Ha, that's amazing. The SEO equivalent of duct taping a colleague's phone or flushing a housemate's toothbrush. But kinda worse.
You said: (yes I know Google doesn't go beyond ~+100 links per page...) | I thought it was 250 per page? Oh boy. I know i need need to do now.
Great article. I loved the video!
It is but 100 is recommended by many SEO's.
Sorry - Google recommends 100 however they will actually index over 10K +
I wrote about this back in January: 100 links/page is really a rule of thumb. It's still a decent one, but on high-authority sites, Google will definitely crawl more links (there probably isn't a hard limit). The SEO trade-off is that more links means less power per link. It's a balancing act:
https://www.seomoz.org/blog/how-many-links-is-too-many
Yeah I read on an official blog post or something like that earlier that it's a case that they didn't used to crawl a huge amount but now there isn't really a limit however if a site is just plastered with them then other factors will kick in ;-)
Yeah, it used to be just a technological limitation. They didn't have the processing power to index over a certain number of Kilobytes of any page (can't recall the number). Now, it's more like the indexation cap. A high-authority site can easily have 500 links on a page (not sure I'd recommend it, but they'd get crawled). A new site with no links might crap out before 50.
Thanks, Rounders is a great movie...especially if you are a poker fan like I am. If I were to guess I would guess that number changes based on the quality of the page/site/etc. just use 100 as a rough guideline.
Chad -
I told you that I was going to troll you...
But this post is just too good for me to do that to you. I think your experience here is a fine show of SEO work. I especially think the social shares was a great idea and a quick reaction. I'm curious, which social networks did you do this on?
Also, I like the steps of resubmitting your sitemap and then manually asking Google to revisit a page. I bet that was key to getting back into the index.
Well done sir. Well done.
I'll troll you on your next post ;-)
Glad you got your rankings back. I was wondering though...how do you know that you needed to do all of that stuff? Isn't it possible that simply removing the noindex tag would have done the job for you? I mean, you've likely got existing backlinks so Google would easily find your site and recrawl it.
Or am I missing something?
I was going for speed of re-entry. Those pages get recrawled every 5-6 days on average, but in many cases we were able to get back 3-5 days quicker than expected. But yes, it wasn't a scientific A/B test.
Not that this is the type of thing anyone in their right mind wants to do a scentific A/B test on, at least on their money site! =]
Haha, yes indeed.
Makes sense...thanks.
It's funny, but I hate that when something like this happens we all have to wonder " Have we been Panda'd"? Thanks for the detailed post. All of us will face adversity and it's posts like this that help us be somewhat prepared.
Great read. It's interesting how quickly you can recover - especially with the correct strategy.
Thanks for sharing.
Google likes active people! This is really good and surprising for me. How chadburgess recover in short time? I was quite unlucky with my website.
Good work! I really like you approach of not get confused of what happened but come up with the quick plan and start immediate action on it!
Even in the normal state we should communicate and participate in community and help other people when needed and this act of communication and connection really help us in the state of emergency!
The post on SEOmoz about xml sitemap was very helpful and I used it for one of my client as well (very giant business portal) and this produce a great impact on indexing… Thanks to Kate Morris for that guide…
Good article! I appreciate that!
I hope it never happens here but I am sure it will at some stage. Great stuff useful for general indexing problems too.
Thanks for sharing
Sheesh, man, I can only imagine the feeling in your stomach when you saw those noindex's, I've been through something similiar and had the entire team starring at me in a boardroom, waiting in anticipation for answers and ETAs.
Thank you for the analysis Chad.
Nice strategy you people applied for getting the pages once again indexed, and that's the similar situation which I faced during my Site development and publishing... well, you have given a detailed strategy report which everybody loves to read. thanks for the post.
Please help!Overview: The official website for the LuxLive lighting exhibition (www.luxlive.co.uk) that will take place on 9-10 November 2011 in London at Earls Court Exhibition Centre. Have a WordPress based site. robots.txt is in order, sitemap submitted via Webmaster tools, all title tags are correct, no noindex tags. Google has 20 of our pages in index.
Problem: However, we DO NOT rank pretty much anywhere organically when someone types "lux live" or "luxlive"; rather, Google shows not that relevant results in top SERPs. We do run a PPC campaign with Google.Can you help? Is this a bug in Google's algorithm?
This thread is probably not the best place for this - feel free to get in touch directly with people for help or ask on Google Webmaster Forums. Consider: When did you submit the site to the index, have you got links and social media shares? And have you got canonical tags? I know sometimes WP can generate duplicate content by accident.
hey Chad, great experience-sharing, thank you for your candid approach to writing the story ! I work for a large .org and have been on your boat a few times but have to admit I have never put into practice the social sharing tactic element as a means of getting re-index and back on the rankings quicker.
Great to see too that you are given thumbs up to talk about it publicly in a forum like this, I envy that. Again thanks for sharing ; )David
Interesting post. It was a nice comprehensive strategy you came up with to get your content re-indexed quickly. The good news from our experience is that content is quickly returned to the index even with just removing the NOINDEX tags. A friend had this issue with their WordPress blog "The Social Network Company" (https://j.mp/Google-TSNC), as there is a privacy setting on WP installs that sets a NOINDEX tag, and is easily overlooked. Un-setting the Privacy setting removed the tag, and his site was fully indexed in days.
Hi there, nice post Chad!
Certainly helpful for fighting this kind of accidents, sure lots of collegues will practically appreciate your experience.
I have some experience with indexation and found some common points between your and my experience, I think someone at SEOmoz would call this a correlation :)
I do hope others can learn from curiosity rather than the hard way, in this last case having read your post will be a relief. On the other hand hard lessons stay longer in the brains...
Thanks again for your time and good luck us all!
I have done that twice now. Once on my dad's horse racing blog I accidentally put noindex on it and wondered why only the RSS feed was indexed. Then on my main blog I decided to block googlebot. I was running a firewall in wordpress and googlebot followed a dodgy hackers link to my site and I blacklisted it without checking. Wondered why Google was dropping all my pages. Both took a lot of stressing and analysing before I worked it out. In fact, an SEO friend saw the noindex for me!
Thank you for this. Will help me sleep tonight. Did the exact same thing a few days ago. Same steps to get it back into index... now the waiting game. Over 500,000 pages gone from the index. o_0Thank you for giving me peace of mind.
Super article. Just gotta be patient now
Hello Chad
Actually I accidentaly deindexed my website by url removal request. Would you please suggest me that how cai i recover it. Kindly help me to get my website back soon
Great stuff for general indexing problems .
I just had this happen to me. I made a php call to open an exit popup window from another page that I did not want indexed (noindex nofollow). Since the call was on every page it no indexed every page that some one went to and the exit pop script poped up on. GEESH! I am waiting now to see how many days it takes to be reindexed. Good deal of information in your article, Thanks!
Hard bur very powerfull practise.. appreciate, tahank You!
Great post, love reading these real life SEO dramas...that was a real nail biter!
I couldn't have found this post at a better time. Having just done a big relaunch for client I thought I'd done everything right, new sitemap, 301 redirects, html sitemap linking to old pages so google picked up the redirects quicker then realised I'd left no index on all pages!!
I had just removed the no index tag, resubmitted the sitemap and manually added the home page back into the index then found this post. So fingers crossed everything will work in a day or so and I'll be as lucky as you and the client doesn't do any Googling over the next couple of days.
I'd like to add an update to my previous post for anyone else in a similar position but i'm pleased to say that the pages are now being indexed and the home page is appearing back at the top of google where it belongs!
Anyone in the same situation follow the steps in this well written post and be patient. The worst part is waiting for the pages to come back but they do and I found in the same spot in the SERPS.
I put the no index in on a wednesday, noticed on the sunday and added the fixes and they were back in the index the following thursday morning. So 4 painful days but drama over and some more knowledge for my arsenal!
Hey Chad - Good work!
What tool do you use to track and graph your SERP's?
Thanks a lot!
I had this problem about six months ago except that the whole news section was blocked by robots.txt (a developer presumed that because I didn't want a page in the sitemap, I would also want to block robots from crawling).
The big difference between noindex and robots.txt is that Google caches the robots.txt once a day, and nothing I did could get the robots.txt recached.
Rankings returned the next day when the robots.txt was next crawled.
I wouldn't read too much in to the return of the rankings because it was the news section of the site that got blocked and most articles only had a 24-hour shelf life, so all I really needed was my new articles to get recrawled.
Chad,
Thank you for sharing the story. Awesome recovery execution, too!
Jay
Thanks Jay.
Too many people forget about the importance of the Sitemap.xml. Good tips and post.
Thanks for sharing this case study Chad! This is the kind of post that's especially useful to bookmark and remember, so when you run into this kind of issue, you know where to start! :)
This is strangely relevant to me. As of last week I had this exact problem with one of my clients. Here's the results:
https://i54.tinypic.com/15mxy7a.png
Interesting. Why do you have big drops in there before getting deindexed? I actually seen this when clients pay for traffic.. i.e. facebook or something else. I can imagine that phone call with a client who monitors their traffic and freaking out! :)
The client is an ecommerce site that mostly deals with government spending and obscure part numbers so the traffic generally dives on Saturday/Sunday.
Most business-to-business sites have that kind of consistent peaks-and-troughs experience with traffic.
Thanks for posting - always great to see case studies. Glad you got re-indexed so fast. Anecdotally, it feels like Google will reverse a NOINDEX much faster than a bad canonical or redirect, but you absolutely did the right thing - in these situations, do everything in the book to get Google interested in those pages.
Thanks for the comment.
Yes, I remember your post https://www.seomoz.org/blog/catastrophic-canonicalization and knew that was at a whole different level of problem.
My theory is that a NOINDEX is different because it's page-specific. If you remove it, you're basically saying "Ok, index me again." A canonical or 301-redirect involves two pages. If A and B still exist and you remove a canonical from A to B, Google wonders "Well, B is still there, so maybe I should still treat it as canonical". The only way to reverse it is to canonical/301 B back to A, which is often impossible or undesirable.
Chad thanks for sharing your experience. I had a situation like this recently with one of my projects, I know the feeling!! Immediate action si the key in this cases, well done!
What impact do you think the blog posts had - was it purely the new page activity linked to the other activity telling Google that you were actually alive?
Google Indexes our blog in near real-time. So my thought process was that I could direct the crawlers back to the pages that we had now removed the no-index tags from.
Of course I suspect (and someone probably can chime in) that a search spider would recognize the location of that link and "know" that they don't need to recrawl it just because there is a new internal link (i.e. they had crawled it just a few days prior). I was trying to overcome that by supplementing the posts with social activity, etc. to send the "signal" that something significant had changed on these pages.
Does that make sense?
Yes that does make sense and I saw after a quick glance at the posts that you had the internal links as well. I guess the off page social activity also helped.
Nice work though blueprint for future activity. I guess large sites need to develop some mechanisms to falg this in realtime - if your servers went down you would get paged - ther should/may be similar tools for flagging this type of drastic revenue affecting change.
We had a great YouMoz post a while back about Google Intelligence Custom Alerts that can provide some ideas for how you might set up alerts to show if your traffic has suddenly dropped off.
Thanks, Chad. Good information to have handy.
I have detected same issue for my eCommerce website and resolved with multiple XML sitemaps and Google submit. This is really interesting case study to work on no index pages. Many SEO guys are facing this issue due to panda update and may be help them to resolve it. Here, I found very interesting thing about social media sharing. One another great use of social network.
Chad thanks for such a great and brilliant post your every word in the article is very helpful really this point should be taken seriously many of the us usually never ever thought into this direction when they facing the downfall in ranking. Well thanks for explaining and sharing such a ethical point.
Very detailed account of the situation - I appreciate that!
Thanks.
I had a virtually identicaly experience back in May of this year. The site had about 2-3k visits a day and it simply dropped out. However, I thought it might be worth noting that we did absolutely nothing except remove the noindex - no link building, no social signalling, just a simple restore of the site to its former glory. Within days the traffic had gone back - to me this was an impossibility and I genuinely believed we had wrecked years of hard work.
My simple conclusion (and this post reinforices it) was that Google has memory. And that Google will forgive you for effectively deleting your site.
Great post!
Links, Social signals, pings and other signals could lead to faster crawl and re-index. If getting back on Google is not time sensitive, perhaps not doing anything (besides removing noindex) would be good.
Great story, that definittly re-emphesises the importance of haveing mulitple sitemaps!
Also, I read an article that Google will ignore a noindex tag on a page that has been +1 (basiclly, they reserve the right to index the page, because they might need to pull a snippet for the exerpt on Google+ when a user +1s your page and shares that page with other Google+ users).
As a insurance plan, we are encouraging people to plus one our pages, and if we ever accidentally no-index a page, the Google+ influence might help stay any negitive effects.
I didn't see that in this case. Many of the pages that left the index had previously been +1d. Interesting though.
I've just done a permanent redirect from one domain to another for a client, but some pages have been left in the index - could this be because of +1's then? That being the case all of a sudden the control of indexation is being left completely to G, which could end up to more duplicate content and a de-rating which is not the fault of the website itself.
What do people think?
Nice one with the +1 Rescue Mission with friends+family... Greetings from the ticket industry in Poland!
Well this is good information to know. Hopefully I don't get de-indexed anytime soon it would probably take me longer than 3 days to get it back up considering man power and everything is low where I am.
Hey Chad,
Thanks for the detailed account. We recently went live with a new website, and followed pretty much the same steps you did in order to get our site of ~500 pages indexed as fast as possible: (1) multiple XML sitemaps (2) manual submissions in Google Webmaster Tools (3) Write blog posts that with deep links to important "hub pages".
The only thing we didn't do (aggressively) was the social linking. I really liked your Google docs approach, to co-ordinate getting your colleagues/friends/etc to +1/Like/Share/etc the important pages on your site
Two thumbs up :)
It's taken me a few days to get round to reading this, but the title caught my attention. I have considered setting up a network of people willing to share relevant content/new websites to people who may have been interested, but I don't know if this isn't a bit 'black' for great SEO? I'm not criticising and I always share my clients' content, but could it appear 'unnatural' to G?
Otherwise, very quick thinking and well done for such a great turnaround :-)
I think it could be unnatural on a systematic basis if the same people were +1ing pages over and over again. But I don't see a problem because really friends and family should be the a core set of the people that like your content, naturally...
Curious to know if other people read that as being a bit 'black' or gray.
Yeah, I suppose so. I wasn't accusing you of 'black', by the way, I was more curious as to whether G could perceive it as being so. I think personally it's a great idea and I wish I'd had the thought myself!
Thumbs up for youuuuu <;)
My heart was pounding while reading this. Great post!
Thanks everyone for reading and having nice comments. But no one gave SERP a DERP the credit it deserves. Add it to your lexicon... :)