There's much talk and confusion in the SEO industry about penalties. We're always seeing in Q&A questions like:
- Do I have a penalty?
- Why do I have a penalty?
- How can I get rid of my penalty?
Recently, however, we've seen more and more examples of algorithmic, keyphrase-specific, page level penalties. Let me explain each of those one by one:
Page Level
Whereas previous penalties were commonly site-wide, harming all pages of your site, we're now seeing page-level penalties being applied more and more. Often these page-level penalties are to the homepage of a site. I'm not sure if this is because most paid links are commonly directed at the homepage or perhaps they're just easier to detect for the homepage of a site. I'm not sure, but we see them commonly on the homepage.
A page-level penalty only affects the page in question and doesn't affect the overall site (though I suspect that it will harm your trust rank somewhat!).
Keyphrase Specific
For me, the most interesting aspect to these penalties is that we are seeing them ONLY applying to a specific keyphrase. This typically happens when there are obvious and easy to detect paid links pointing to a page and they all have exact anchor text for a particular keyphrase. This can make the penalty particularly difficult to detect since the page can continue to get search traffic for long-tail phrases but not from the particular keyphrase you're going after!
Algorithmic
This is more of a consequence than a specific requirement of a penalty, but in my opinion these penalties are being generated algorithmically. We've seen anecdotal evidence of a penalty taking effect, then not taking effect as the page selling links get crawled and re-crawled. Sometimes Google finds the paid links and sometimes they don't. This would make sense as well because if you're going to hand out algorithmic penalties, you want to make sure that you don't screw things up too badly when you get it wrong (and they will get it wrong from time to time). By limiting the penalty to a specific page and a specific keyphrase, they ensure that if they needed to they could apply a penalty algorithmically to a page on the BBC without penalising the whole site. Equally, the solution is often straightforward - remove the offending links and wait for a re-crawl from Google.
These new penalties (which I don't think are new at all, but we're seeing much more of them now) mean that the above flowchart doesn't work so well. Algorithmic penalties are much harder to detect - after all, people are always saying they 'deserve' to rank for a particular phrase when in truth they just don't have the right number or quality of links. So how CAN you detect a page-level penalty? I've not got a fancy flow-chart as this process is still being formulated, but here are a few things that might tip you off to an algorithmic page level keyphrase-specific penalty.
How To Detect Algorithmic, Page Level, Keyphrase-Specific Penalties
- You have paid links which are from a link network and would be easy for Google to detect algorithmically
- The paid links all the use the same anchor text and point to the same page
- The page being linked to doesn't rank well for the target keyphrase. Watch out for a -30 style penalty; if you're ranking top of the 4th page consistently no matter what you do, you might have a -30 penalty. These can come in other forms too; I think -5, -10, -999, etc. have all been seen.
- The page in question DOES rank ok for closely related keyphrases, which shouldn't alter the search results much. For example, if the keyphrase you don't rank for is "dog grooming" but you DO rank for "dog grooming www" and "dog grooming .com," then this might be evidence of a keyphrase level penalty. Watch out for keyphrases like "dog grooming services" though, as you might rank better for related keyphrases due to a higher number of anchor text links. So make sure you test across a couple of keyphrases.
PS - In other news, Google Streetview has come to London! And there's an interview with the Distilled SEO team, which I think would be interesting to readers.
Hi Guys,
Im a long time reader, first time commenter, and full time SEO for a blue chip for 3+ years.
page level penalties have been in place for some time now, the first page of mine that I noticed it happening on was around october 21st last year, since then four other pages have suffered the same fate.
In my specific industry its incumbent on me to rank for a specific keyword on an exact date, ie. "blue widgets" on the 31.12.2009 - we are normally in or around 4th-6th for every related term and on the run up to the date I normally chuck a load of internal and external link strength at the pages, with several hundred related anchor texts from related sites - the page penalty phenonmenon has been happening since October and has happened on about 15-20% of the campaigns I have run since then.
The behaviour however is very interesting - the pages will rise throught the SERPs over a 2 to three day period, and maintain the #1 slot for 5 to 7 days before being hit with a -30 penalty.
I have in the last week worked out a great workaround that means it takes about 48 hours to have another page put in its place - but given the trustrank implications Im not going to rely on this procedure.
Hey MOGmartin, thanks for commenting! That sounds like a fascinating process. I'd love to hear more about it either here, on youmoz or privately on email?
Yeah, the page level penalties have been around for a while but we've seen a sharp increase recently which is what prompted this post.
Hi Tom,
thanks for the reply! Im fine discussing the issue in the public arena, the workaround is kind of priveleged information however - and its not something that I would advocate people doing, I just came up with a theory, ran with it and it worked a treat!
If you want any specifics on the linking or diagnosis be sure to let me know,
MOG
I'd point out that we've seen 301s working quite well for this kind of thing. Not ideal, but when you're in a bind with a penalty on a page level, they can seem to "clear the slate" so to speak. This is not a new tactic of course - been around for years - but if you're pretty sure you didn't do anything wrong, can't ID the problem and have good trust/authority/etc across the rest of the site, it can be worth a try.
Yup, 301'ing the old page was part of what I did to retrieve the serps back to a new URL, mixed in with some creatively adjusted links to and from...
My concern would be having this tactic flagged to google, or just hitting a threshold of on site pages that have been penalised.
Just wanted to add that your experience mirrors ours, Tom. In Q+A the last 2 months, you can see a really large rise in the number of sites that have been penalized. We've even been doing some triage work on some big brand and well known startup sites that unknowingly violated guidelines (even though they haven't even engaged in true "SEO"). I think the dials for a lot of things - on-page stuff, excessive internal linking, external linking, etc. have bee turned up (maybe not all at once, but the cacophony makes it feel that way).
Thanks for the post!
perhaps this could go in the beginers guide that rand is doing?
Often a key thing that people worry about, and very nicely explained (especially the flowchart) - I've yet to experience any penalties - luckily But it doesnt hurt to know how to check/react if they do happen!
ps. Streetview - took em long enough!! But very happy - and think its better quality imagery than the US?
Edit: Actually my reaction after going thru the flowchart would be to get in a v.good agency to help me out, as at that point the benefits of inhouse getting in outside agency help completely makes sense!
Yeah, penalties are really hard to diagnose without expert knowledge. A lot of beginners *think* they have a penalty when what they really have is a lack of links. Conversly some people can't figure out why they can't rank and it's because of a penalty.
Especially when Google is shifting the goal posts they can be really hard to figure out.
"A lot of beginners *think* they have a penalty when what they really have is a lack of links"
Or none at all, or a very search engine unfriendly website :-)
haha, well a nice flow chart's a pretty simple and effective way to guide them through that process!!
... and a lot of them take it even further and think they have been banned and boy is it difficult to persuade them sometimes!
I think it's a bit too advanced for a beginner level - but indeed, one of the best explanations of penalties I've ever seen.
Yes I agree with you, it is bit tough to analysis for a beginner.
I've had a very interesting experience with a new client six months ago:
The client was not ranked for its own url, site: or any other method of finding a site on Google - his site was obviously banned from Google.
It was an ecommerce website that suddenly lost its ranking and the client asked for help.
I looked at the links to his web site and found ~10 links pointing to his domain. When I went to check these pages, I found pages with no content(or minimal content), with many text ads on related subjects.
I viewed the source and discovered that they were all from Yahoo! content network.
Then I tried to find those linking pages/sites on Google and couldn't find them as well.
Apparently people are buying domains, put ads from Yahoo! (the client was using Yahoo! search marketing) on pages with almost no content. Then, Google finds these pages, mark them as SPAM and all the other pages that they've linked to - marked as SPAM as well.
BTW, The client didn't have any "good" link pointing to his site.
The client was naive. He was not engaged with any SEO services (no tricks, cloaking etc') but has been banned from the Google Index just because he advertised with Yahoo!
We switched domain name, added one good link and BABAM after 2 days he was in the index again.
I think this is an interesting experience and wanted to share it with you.
Tom and Rand, I have seen this happen over the past year on a few occasions and on different domains. The first instance it was an internal page that just dissapeared in the SERPS after climbing steadily and at it apex hitting page 3. Fortunately we export backlinks from Google webmaster tools bi-weekly and archive them. So we were able to isolate the drop in the SERPS that coincided with a link that was obtained around that same time. The link was mistakenly put sitewide when it was not supposed to be. Removed link, started ranking again immediately. Viola!
Second case, and most recent, is much more perplexing. Dealing with a homepage that is ranking for all other terms extremely well and was ranking #3 for the main term then it dropped -133 momentarily (the report auto ran at 12:24am and by the morning it was only a -8/-7).So its currently at #10. <scratch head>But its ranking so well for one-word keywords that are SUPER competitive as well as 2,3,4, etc word phrases all related. So its just that one kw terms.</scratch head> That one term does command a very high count when it comes to inbound anchor.
I guess I just have to wait and see what fate Google doles out.
Hi Miguel-
That's a great idea to export the backlinks weekly! We've been tracking them, but not with that regularity. Thanks for the great idea. We've had some problems with competitors spamming us from *horrible* websites to drop us, which has worked for them, but makes it hard to track for us. The weekly export comparison seems to be a great way to stay on top of this.
Thanks!
great post Tom, would be great to see Matt Cutts' thoughts on this. As you know we're battling with what looks like an algorithmic penalty.. we're mid clean up and will let you know when the re-inclusion request has been submitted and the consequential results.
I'd be interested to know if anyone has any experience of sucessfully removing links from the digital point network now that it is closed.. or other spammy link sources that ignore the more obvious means of communictation, do cease and desist orders work with these types of spammy networks or do they just ignore them?
I would expect C&D could work if you can identify the person to serve with them. Even though I'm not sure you can actually stop people linking to your site, many people back down from legal letters etc. Could get expensive though with a big network!
thanks will.. we're going to try all avenues so fingers crossed
I don't know of any legal leverage to stop a site from linking to you. I've seen plaintiffs (wrongly or rightly) assert that given the context, the link is a trademark infringement and/or false advertising and/or copyright infringement. All of these cases are brought by brand holders who argue that the linking site is misrepresenting or exploiting their brand or content in some way.I don't know of any legal rule that can stop a site from linking to you if there aren't trademark or false advertising or perhaps defamation claims also present on the linking site.
Generally, I think the lack of regulation about who can link to who is one of the best things about the internet! I'm all for the unhindered spread of information. However, in this case, it certainly seems unfair that you cannot prevent a spammy site from linking to you if they aren't abusing your brand, copyright or reputation in some legally cognizable way.
You can write a cease and desist letter and hope it gets some traction, but in the end, I don't think you could follow-up your letter with a winable lawsuit. I suspect we'll continue to see more litigation on this topic in the future.
Best of Luck!
We have a client here in the UK with this problem. They've been linked to by some spammy sites and we can't seem to shake off the links.
Sarah makes an interesting point about the legalities of someone linking to you unless you can PROVE they are causing a problem, and how hard can that be sometimes!
Regs....David.
One of the oldest questions in the book this one Tom - it's good to have some pointers about what to look for, especially when the only other option is banging on the Iron Curtain...
We've (as you know) beenwrestling with a potential penalty on behalf of a client and I'm very interested to see if we get ant kind of response from Google on it.
Plus on Streetview - brilliant! Griff has already found a crimewatch photo of me through the office window!
"dumb question, how do you even know if you have a penalty?"
I don't know. But at this point it is all I can assume.
Since you're an expert what is your opinion?
I'm not an expert on this particular topic, I guess that's why I'm asking the dumb question! :)
I see!
I think I misinterpreted your original comment. Sorry about that.
dumb question, how do you even know if you have a penalty?
Hey there,
It's never a dumb question, especially with the page-specific penalties that are being discussed here. Unlike the site:url search that will let you know if you're banned, there is not an 'easy' way to know if you've received a penalty. There are several ways to figure this out, however:
1) If you suddenly and dramatically drop in traffic. This can be seen by looking at your historical traffic data. Hopefully you're tracking page-by-page data and that will tell you if you dropped dramatically.
2) Check how dramatic of a drop you had and whether it coincided with a Google shake-up. Chances are that if you have a huge drop at the same time as a shake-up you've gotten penalized for something new.
3) Like they mentioned in the article, compare yourself with others in your SERP verticals. If you're the only one now missing from the SERP, then you're likely penalized. If everyone got shooken up and newbies are now in the top 10 of your search, then it was probably an algorithm change.
Usually, if you've got basic good white-hat SEO going on on your end and you see a huge drop in traffic or SERP ranking, you probably got a penalty. The real trick then becomes to figure out why you were hit (duplicate content? 301 issues? etc.) and where to go from there to regain your standing.
Good luck!
This is such as interesting thread. I registered just so I could comment here.
I have a website where the home page has been messed up for a very long time now (over a year).
Here is the problem:
Take a unique phrase from the home page, put double quotes around it, and google it. You will not find my page. You will get no results found.
My home page is a PR4 and I am sure it is indexed. I have no clue as to if I am penalized or not. I don't think I did anything to get penalized. I have sites that are very similar from an seo perspective to this site and they have no problems whatsoever. The deep pages of the site actually get a very generous amount of traffic from google, just not the home page.
The page ranks on the first page of the other engines (live, Yahoo) for my primary keyword.
Just can't figure out this google problem for the life of me. Like I said it has been over a year now. Driving me crazy to the point where I may just redo the site with a different domain.
By the way, see www njshore-rentals com if you want to see the site I'm talking about. If anyone sees why this could be happening, I would love to hear what you have to say.
Carl
This is a great topic as I've been in many situations where paid linking has been a problem, affecting some verticals more than others and of course on a keyword or theme basis.
The real kicker is that because the issues are so subtle, the type of data to be collected becomes very burdensome...
In my opinion there should be a way to formulate a page-specific or keyword-specific detection method, to at least get within 70% accuracy (better than anything now). I am sure as more of us put our heads together on this, a clear methodology will be appearent.
I'm sure the moz guys can correct me if I'm wrong but I think there's a tool coming out soon which will help analyse the top 40 or so sites for a specific keyphrase and reverse engineer the algorithm, that could easily have applications for spotting this kind of discrepancy...
I was going to add that to help with detection don't just look at your own site, but also consider those others that are appearing around it in the results. Consider if there is a lot of churn anyway in results over time or are there other sites that have appeared and pushed yours down. Look for the trends; here's where keeping a hold of all that ranking data comes in handy!
As for street view, surprising coverage for Dundee and Aberdeen, not only the cities but half the respective regions as well!
There are some really interesting responses in here, I'm glad to see some of our own theories on a workaround being validated as well.
I'm very interested to see how Google push the penalties forward - if algorithmic penalties are affecting websites who don't even know they are doing something wrong they may start to step on some toes.
Although as a result I guess it means more business for us. However my point is that while the 'carpet bomb' approach is useful for obvious spamming it seems that the penalties either need to:
(a) remain algorithmic but become much more complex to avoid catching the wrong sites
(b) at some point be sense checked by a human to ensure the wrong sites aren't caught
I would also (as I'm sure many other would) like at least some clarity on penalties from Google. As usual all they seem to do is preach the gospel (design good sites for people, read the webmaster guidelines etc) and completely sidestep direct questions.
I'm not asking for a blow by blow guide however at the very least confirmation that your site has been penalised would be nice.
"(a) remain algorithmic but become much more complex to avoid catching the wrong sites"
See my story above. For some reason, I beleive some of the wrong sites are already being caught!!!
Somehow my comment went in twice. Sorry. Deleting the dup.
We're reading here that penalties can come in the form of -5, -10, etc. With such a vast range (almost any number), it could be that we just can't figure out why we can't improve rankings beyond that point and blaming a "penalty" too :P
This is a seriously good chart on many levels. The biggest benefit I see is that it takes something that is today a HUGELY emotional issue because of the fear out there, and turns it into a left-brain exercise.
For those managing SEO for clients, you know what I'm talking about. There is just so much fear running around. It's there for many reasons.
1) People are scared of things they don't understand
2) There is a ton of BAD info on the Internet about Google bans
3) They understand the serious consequences associated with losing rank on a key term.
Yet - They need higher rankings. Almost like a drug adict, they need to move higher and higher...
Sorry - I digress. This is a great chart to use with anyone who is freakin' out to give them a framework to put the issue in perspective.
This is crazy.
These can come in other forms too, -5, -10, -999 have all been seen I think. What kind of impact does the fore mentioned penalties have?
thanks for the post.
This numebr refers to the position drop caused by the penalty. For example a -25 penalty would cause you to rank 25 places lower than you would without the penalty.
thanks.
-999 is not correct. That is the 950 penalty. But doesn't matter anyway.
If you can get banned by what that flow chart shows, why don't i spend time banning other sites then? It's just algorithmic filters, unless Cutts or one of the Indian keyboards manually bans you. Like if you get 800 new links, yet traffic doesn't rise it's dubious so triggers a filter. They could identify spam nodes however but would take tons of time to evaluate.
Some people do, unfortunately, do this. I think the problem is it's difficult to know what will get them banned and what will help them rank better.
Google work quite hard to stop this kind of thing working (i.e. removing competitors from the search results) but I suspect they've still got a lot of work to do in this field and 2009 might be a year it becomes more prevalent.
Prevention is better then cure so keep the basics right.
If its difficult to find the right disease (penenlty) then why not prevent them from happening.
I know many times the lack of knowledge will get people banned but thn you have to do alot of hardwork to get ban lifted and you have to find why penelty was laid so why not do it before.
What is wrong with your term target tool? It's been giving me errors for 2 hours.
We were experiencing a bug. It should be fixed now. Let me know if you are still seeing problems. sitesupport at seomoz dot org
GREAT article, I'm bookmarking it for future reference. One of my websites has an exact keyword URL. It's been live for 2 years and still does not even rank on page 50 for this particular keyword. I've been building natural backlinks and everything - and the keyword in question only has about 650k results in Google. I was never able to figure out why.
What is the gTLD?
Great article...I'm working on a site right now that seemed to be suffering a penalty only on the homepage. When I first starting working on it, the page had been dropped from the index and its PR had dropped from five to zero.
I found some hidden text on the page (a sloppy CSS conflict resulting in black text on a black background), as well as a few variations of the homepage floating around in the SERPs. Once that was cleaned up, the page regained its PR 5, but was still excluded from the index.
I thought that we might be suffering some sort of penalty, but I couldn't find any information on page-level penalties!
We're rebuilding the homepage now. Hopefully, it will bounce back.
I am at the moment trying to rescue a site that's been handles by a young and overly ambitious SEO. Essentially the guy has gone nuts with keyword repetition, placed it in every SEO relevant tag and bolded every single one of them. The site was never able to rank for anything better than page 4 and its competitors are doing better with the most ridiculous of the websites. Beware and don't go too far with your on-site SEO.
Damn Rand, Title Case Much? Maybe Give Your Shift Key And My Eyes A Break Once In A While.
Don't blame Rand for my titles, this one's all my fault :-)
Apologies for the capitalisation, I've been writing Reddit headlines far too much recently.
I title-case all my titles for the blog too - I think it's correct behaviour (the clue's in the name).
Excellent Matt-Cutts-bait! So, where is he?