Over the past few months, we’ve seen a number of public statements from Google regarding algorithm updates that sound something like this:
Notice the carefully chosen phrase “X% of queries noticeably affected”. Those statements sound very specific, but once you stop to think about them, you’re inevitably left with a couple of questions: (1) percent of which queries?, and (2) exactly how much is “noticeably”?
Percent of Which Queries?
When most of us hear “X% of queries”, I think we naturally assume unique queries (different keyphrases), but there are really two interpretations: (1) unique queries, and (2) total query volume. Here’s an example, from a conversation I posted on Google+ - let’s say that we live in a world where there are only four searches anyone ever uses:
- “iphone 5”
- “justin bieber”
- “platypus pants”
- “charlie sheen”
If we ran 100 searches on our parallel universe version of Google, the volume would be distributed to our four keyphrases as follows:
- 50
- 40
- 9
- 1
That's right - suck it, Charlie Sheen. Now, let's say that query (c) is impacted by an algorithm update. By definition #1, this would affect 25% of queries. By definition #2, that same algo update would only affect 9% of queries.
I’ve put this question to Google a couple of times, and while they didn’t reply directly, a few people were kind enough to repeat it. Jacob Klein’s tweet to Matt Cutts got the following reply:
It’s not quite an official statement, but I’ll take it. In most cases, Google is talking about overall query volume. I wouldn’t read too much into the “typically” – Matt naturally doesn’t want to commit to “% of queries affected” only ever having one meaning when uttered by any Google employee. I’m comfortable, though, that Google’s recent statements refer to query volume. This also makes sense from the way Google views the data – overall volume is probably much easier to measure than unique queries.
I don't think Google is being deliberately cryptic in this particular case – the statement is simply ambiguous. As SEOs, we naturally think of "queries" in terms of unique keywords, because that's what we track. Google thinks in terms of overall search volume and each query is a discrete unit..
How Much is “Noticeably”?
So, now that we know we’re talking about query volume, what’s this “noticeably affected” bit all about? Does any change count, or does it have to be significant? Hat tip to Matt McGee for reporting on a follow-up conversation that sheds some light on this one:
It’s not quite cut-and-dry, and the definition of “noticeable” may be a little complex, but basically they’re talking about Page 1 changes and probably even flux in the first few rankings. A query where the #32 ranked keyword drops to #34 isn’t going to be counted as “noticeably affected” by an algorithm update.
Knowing Is X% of the Battle
So, when Google refers to “X% of queries noticeably affected” they mean the total volume of queries and a significant change in the Top 10. Since we don’t see the entire universe of queries that they do or really know the relative volume, this still leaves a lot to interpretation, but I think any little bit of transparency helps at this point. It’s a useful gauge of relative impact as Google uses the same metric across multiple updates.
Keep in mind, though, that any given query either impacts you or doesn’t impact you. Don’t rely on the aggregate statistics – pay attention to what matters to you. Look at it this way – unemployment was just under 8% in the US in September, but you and I either have a job or we don’t. That 8% may be a useful gauge of economic prosperity or the effectiveness of our leaders, but ultimately we have to be aware of our own situation. If an update only impacts 0.3% of queries, but your money term was one of them, knowing that 99.7% of queries were untouched won’t be of much comfort to you.
For reference, I've recently added Google's impact percentages to the MozCast Events page. This page lists recent algorithm updates, along with the severity as measured both by MozCast temperature and Google's publicly stated percentage, to allow for easy comparisons.
I personally ignore what % of queries was affected, Just jump into testing and check all my keywords that are on their same position or not. I encounter a client whose whole business ruined just affected in 0.1 % queries all of his top ranked keywords were nowhere in top 50s. So why we bother it, just wait about some data to come out and start working to recover from update.
Had a similar experience with a recent client as well. We had to drill deep down to the problem around the high-value keyword that they've lost ranking for.
Thanks to Dr. Pete for providing us a better interpretation of Google's weather report.
I want to be sure I understand this correctly.
0.3% noticeably effected means to Google:
0.3% of 1st page SERP's .
What I'm having trouble with is how Matt can say its search volume and not keywords. Does that mean if I search for "fluffy bunny neck ties" 3333 times, one time I would get a different 1st page SERP?
I think so too. Confirm my assumptions please...
I think it would mean if x keywords have 500,000 websites ranking in the top 10, 1500 of these would be effected.
Hmm, yeah. Sounds truly!
Sorry, no - it just means that the % is based on total volume, not the individual keywords. So, if the entire world did 10,000 searches, "fluffy bunny neck ties" accounted for 3,333 of those searches, and it was affected by an algorithm update, that one keyword would massively shift the percentage.
Put more simply, "fluffy bunny neck ties" counts as 3,333, not as 1. If an update hits popular, high-volume keywords, the % will be higher.
Here is a couple hypothetical ways that Google could generate tweetable metrics like "X% of queries...":
1. Extrapolate a sample of live data tofind % of total search volume
They could run a percentage of their searches against both the new and the old algorithm, do a diff to see if the SERP has "noticeably" changed under the new algo and extrapolate from there, reporting when they've achieved a statistically significant sample size. This would be pretty easy for them to rig up, and extrapolating would yield a percentage of total search volume.
2. Do a ton of calculation to find % of change over all keywords
Finding the same metric for a keyword would be a lot harder. If you wanted them to report on the change over all keywords, they'd have to basically wind up calculating at least the first page of every SERP for every query, AND they'd have to keep a historical log of those SERPs as well. This seems extremely computationally expensive, but then they're Google and have vast amounts of servers scattered all over the world specifically for cranking away on problems like these.
Number 1 seems way more likely.
I strongly suspect they calculate this metric during live-testing, which would be more similar to (1). It's not across the entire world of queries - I'm 98% sure of that. I also doubt it's a before/after, because they seem to have the number as soon as they announce that an update is going live.
Live testing would have certain seasonal risks. I'd guess that they see very different popular queries around the holidays or elections than on average. To mitigate this it seems reasonable to cut out a percentage of the queries that would be seasonal, or based on a specific recent event.
Live tests are generally A/B - maybe 1% of search users will see the update as a test. During that test, Google could simply measure change/impact for the experimental (B) group against the control (A) group. I assume they do this anyway, as they need to know the real impact. I doubt they can adequately simulate what an update is going to do in the wild - eventually, they have to put it out there and start measuring.
Good post. I was contemplating this last night in #seopub while chatting with Duane Forrester. I don't think it's a specific keyphrase but rather a specific bucket of keyphrases that are categorized by intent.
For example, we know that those 3 different types exist (transactional, navigational and informational) as well as some new ones per the 2012 Quality Raters Handbook. I think that SEs have a list of keyphrases categorized by intent in its database and when updates come out, they impact those intent categories.
When one takes one intent category and divides it by total query volume then you get a tiny X% number which in reality might be a cataclysmal change if it impacts keyphrases that bring you traffic. When I look at it from Google's perspective; is it smarter to have that X% be a small number or a big number that may result in more negative PR for them?
I feel that it's a PR management technique that people should take with a grain of salt. Regardless of what percentage that number is - we should focus on making sure that the keyphrases that drive traffic to page xyz are correctly mapped to that page in terms of intent.
If its not, then its very likely that one will be impacted by that X% algorithm. Thoughts?
Absolutely - a change could impact a small percentage across all keywords but have a huge impact within a certain bucket, whether it's an industry/niche, query intent, etc. One thing we're trying to do on MozCast is expand the data set sufficiently to break down flux by some industry categories. It could be that average flux is pretty low but that one niche, like real estate, sees huge changes.
Is there a fail safe way to figure out query intent? How do you determine that a word means what it means? For example, if someone searches "photos" versus "pictures." Is there any conclusion that one can draw about the intent beyond the word's definition (such as demographic or psychographical information or which one is more relevant)?
In a word: no. That's why personalization and search history are so critical for the SEs. They're the best clues of intent when you have an ambiguous query. The other clue is overall volume and general behavior. If someone types "apple", Google assumes in 2012 that that's Apple the company. They word alone doesn't tell them that, but the behavior of internet users in general does. If they know that I typed "buy ipad" 1 minute ago, then their certainty is high.
Google falls back on dominant interpretations in many cases, which has pros and cons. This post I wrote a while back about the quality rater guidelines gets into that topic more:
https://www.seomoz.org/blog/16-insights-into-googles-rating-guidelines
Thanks Dr. Pete for this awesome post. I am expecting blog post on this subject from you. you have written with excellent example i am always read your post and waiting for updates from you. It is really challenging to maintain our ranking after algorithm updates but i am happy there is no any effects on my keywords after algorithm update.
Thank you for the information Dr. Pete. Interesting how Google issues these public statements but keep them so vague.Thank you for keeping us informed on Google's algorithm updates.
Matt Cutts replies are ambiguous, cryptic, non-committal? I never!
Thanks for the post, it does make some of these 'x% of queries' comments a bit more clear.
Well, "Basically" and "might" still leaves a lot to the imagination ;-)
Hello Dr. Pete
So what is the final conclusion, what is % of queries which Google mentioned?Kindly advise..
Thanks a ton!!
i believe there is not a simple conclusion. it is Google speak and nothing else
"Knowing Is X% of the Battle" - heh. funny.
When we talk about any google update then here is a true fact to accept that it cause a great change on wide scale so it's very important to deal tactfully to cope up the issue in this regard.The views of renowned persons helps a lot in better understanding the exact update criteria.
No I feel like such an ignorant for thinking it always meant 0.0X% out of EVERY english query!
:\
I was wondering with
how does a change impact " unique queries " ? Would't it make more sense to report on % of affected domains / pages ?. When Google does a change to its algorithm , what ever the search queries the impact is felt with domains / webpages that show up in the SERPS . So reporting on things like x% of the domains have seen noticeable impact ?
Pardon the ignorance ( if any ) , I am just trying to get my head around this ^_^
Google looks at the query as a whole, not at the number of domains/sites effected with that query, at least for these impact percentages. Basically, a query is either impacted or its not. So, the only question is how they count, and that's by total volume.
What we do on MozCast is actually closer to what you're saying. Although it's still based on the rate of change in the Top 10, the more sites that move, the "hotter" the temperature is. So, we try to factor in both. Of course, Google probably has many metrics they actually use. This post is just about the one they report publicly.
thanks for taking the time to explain that :)
Great post Dr.Pete, Thanks for sharing wonderful post with us. Dr.Pete i have one question, Google changes its algorithm in every month( Like Google panda in every month and Penguin between 4 to 5 months). I am 100% sure about that google will continue do this. But my question is that if google will continue update it's algotherm, after 4 or 5 years google will stop SEO?
nice post - and great hypothesis but really this is just a "best guess" of what the meaning of this is.
for me the % variable really brings up some much larger questions that will not be answered for a few months until all you big boy SEO companies do a hard analysis of your indexes.
Google clearly staggered this update just to keep everyone on their toes/throw people off by not creating an obvious footprint and showing direct causation etc.
Thanks Dr. Pete. I'm a big fan and always learn a lot from your posts. I like what you have to say about paying attention to what matters to you. While it's interesting to know from a purely theoretical standpoint what an algorithm change effects, it's really only super-relevant if it effects you. An figuring out if it was the algo change or some other factor that caused you to move up or down is a big challenge too.
There are so many factors that play into rankings as we all know, fixating on the algorithm seems like a futile exercise because it's something we can't control. All we can do as SEOs is work on the things that we can control, do what's right and hopefully that leads us to success.
Good to know for your own stats as well.Finding other related sites and information for keyword results in search.Good for Rating
It is generally about % of total queries Google receives be it from Desktop or Mobile..
Very nice information, thanks. So essentially when they say 0.3% of queries it might actually be more that is effected as noticeable means top 10 results.
The "Noticeably" section was a great one for getting me thinking. I've always been aware that the web is a visual medium, of course, but that sometimes gets pushed into the back of my mind when analysis comes into play. That is, when we start talking about numbers and stats in relation to searches, my "visual radar" shuts off and everything becomes abstract.
The discussion about search results that are immediately visible (in the browser window) having more significance to a measurement of change than results farther down was a good reminder. It's not just the raw change of any arbitrary result that matters - the metrics are more sophisticated. It's not just about the numbers. It's about what the user sees.
Sometimes I wonder why they release these statements at all. It is not like there is any organization out there holding them accountable to the statements or changes they make.
Ultimately, even when they provide clarifications, they always seem to be mired in vagaries i.e. "typically" "Basically". It seems these percentages are totally subjective... similar to their announcement regarding "not provided" - @Dr. Pete are there any tools out there other than your awesome MozCast that would give any sort of indication as to whether what they are reporting is true or relevant?
There are two other systems like MozCast:
(1) SERPMetrics Flux Charts - https://serpmetrics.com/flux/
(2) SERPS.com Volatility Index - https://serps.com/tools/volatility
They're based on different data/metrics, but the concept is similar to MozCast (to be fair, SERPMetrics launched first of the three, I believe). It's still a relatively new idea, and I don't know of tools measuring things like % of impact, because only Google really sees that.
The difficulty is that Google can separate the precise impact from a single update from other updates and the background noise. The background noise is massive - SERPs fluctuate frequently and sometimes wildly. I've calculated that as many as 80% of the keywords we track have changes in the Top 10 every 24 hours. At this point, we simply can't separate signal from noise like Google can.
it really confusing. One more question is exact match and broad match. All the queries related to or include "Justin Biber" in broad match counted as one or different for all instances in final percentage?
It's a reasonable question based on my example, but for overall queries, match-type doesn't really matter. Google is seeing the sum total of the data (or, at least, for some population of queries). As SEOs, we would use match type to try to calculate volume (since that's how we get the data from Google), but they don't have to pull the data that way.
It's nice getting at least some idea of what Google means when they say % change. I think the "definition of noticeably" is more helpful than % because it lets us know if our keywords were affected.
As mentioned, 0.1% or 90% doesn't make a difference to me - the key is if the queries I'm targeting changed. Knowing the signs of a query affected by the change will help to identify which keywords were in fact affected and if action needs to be taken.
This is tremendously useful Dr. Pete - thanks for the help. It certainly helps to put these algorithm updates in perspective.Also, when they say 0.7% of queries, and don't specify English, does this refer to all languages - that vastly dilutes the numbers.
That's unclear - I think sometimes English is implied and sometimes it isn't, unfortunately. It depends on how the update rolls out, I suspect. If it hits the US first, then I assume English is implied. If it's an international roll out, it's hard to say.
thanks for these great post , very informative we need more of that