Like it or not, rank-tracking is still a big part of most SEO's lives. Unfortunately, while many of us have a lot of data, sorting out what’s important ends up being more art (and borderline sorcery) than science. We’re happy and eager to take credit when keywords move up, and sad and quick to hunt for blame when they move down. The problem is that we often have no idea what “normal” movement looks like – up is good, down is bad, and meaning is in the eye of the beholder.
What’s A Normal Day?
Our work with MozCast has led me to an unpleasant realization – however unpredictable you think rankings are, it’s actually much worse. For example, in the 30 days prior to writing this post (10/11-11/9), just over 80% of SERPs we tracked changed, on average, every day. Now, some of those changes were small (maybe one URL shifted one spot in the top 10), and some were large, but the fact that 4 of 5 SERPs experienced some change every 24 hours shows you just how dynamic the ranking game has become in 2012.
Compare these numbers to Google’s statements about updates like Panda – for example, for Panda #21, Google said that 1.2% of queries were “noticeably affected”. An algorithm update (granted, Panda 21 was probably data-only) impacted 1.2%, but baseline is something near 80%. How can we possibly separate the signal from the noise?
Is Google Messing With Us?
We all think it from time to time. Maybe Google is shuffling rankings on purpose, semi-randomly, just to keep SEOs guessing. On my saner days, I realize that this is unlikely from a search quality and tracking perspective (it would make their job a lot messier), but with average flux being so high, it’s hard to imagine that websites are really changing that fast.
While we do try to minimize noise, by taking precautions like tracking keywords via the same IP, at roughly the same time of day, with settings delocalized and depersonalized, it is possible that the noise is an artifact of how the system works. For example, Google uses highly distributed data – even if I hit the same regional data center most days, it could be that the data itself is in flux as new information propagates and centers update themselves. In other words, even if the algorithm doesn’t change and the websites don’t change, the very nature of Google’s complexity could create a perpetual state of change.
How Do We Sort It Out?
I decided to try a little experiment. If Google is really just adding noise to the system – shuffling rankings slightly to keep SEOs guessing – then we’d expect to see a fairly similar baseline pattern regardless of the keyword. We also might see different patterns over time – while MozCast is based on 24-hour intervals, there’s no reason we can’t check in more often.
So, I ran a 7-day crawl for just three keywords, checking each of them every 10 minutes, resulting in 1,008 data points per keyword. For simplicity, I chose the keyword with the highest flux over the previous 30 days, the lowest flux, and one right in the middle (the median, in this case). Here are the three keywords and their MozCast temperatures for the 30 days in question:
- “new xbox” – 176°F
- “blood pressure chart” – 67°F
- “fun games for girls” – 12°F
Xbox queries run pretty hot, to put it mildly. The 7-day data was collected in late September and early October. Like the core MozCast engine, the Top 10 SERPs were crawled and recorded, but unlike MozCast, the crawler fired every 10 minutes.
Experiment #1: 10-minute Flux
Let’s get the big question out of the way first – Was the rate of change for these keywords similar or different? You might expect (1) “new xbox” to show higher flux when it changes, but if Google was injecting randomness then it should change roughly as often, in theory. Over the 1,008 measurements for each keyword, here’s how often they changed:
- 555 – “new xbox”
- 124 – “blood pressure chart”
- 40 – “fun games for girls”
While three keywords isn’t enough data to do compelling statistics, the results are striking. The highest flux keyword changed 55% of the times we measured it, or roughly every 20 minutes. Either Google is taking into account new data that’s rapidly changing (content, links, SEO tweaks), or high-flux keywords are just inherently different beasts. The simple “random injection” model just doesn’t hold up, though. The lowest flux keyword only changed 4% of the times we measured it. If Google were moving the football every time we tried to kick it, we’d expect to see a much more consistent rate of change.
If we look at the temperature (a la MozCast) for “new xbox” across these micro-fluxes (only counting intervals where something changed), it averaged about 93°F, high but considerably less than the average 24-hour flux. This could be evidence that something about the sites themselves is changing at a steady rate (the more time passes, the more they change).
Keep in mind that “new xbox” almost definitely has QDF (query deserves freshness) in play, as the Top 10 is occupied by major players with constantly updated content – including Forbes, CS Monitor, PC World, Gamespot, and IGN. This is a naturally dynamic query.
Experiment #2: Data Center Flux
Experiment #1 maintained consistency by checking each keyword from the same IP address (to avoid the additional noise of changing data centers). While it seems unlikely that the three keywords would vary so much simply because of data center differences, I decided to run a follow up test to measure just “new xbox” every 10 minutes for a single day (144 data points) across two different data centers.
Across the two data centers, the rate of change was similar but even higher than the original experiment: (1) 98 changes in 144 measurements = 68% and (2) 104 changes = 72%. This may have just been an unusually high-flux day. We’re mostly interested in the differences across these two data sets. Average temperature for recorded changes was (1) 121°F and (2) 118°F, both higher than experiment #1 but roughly comparable.
What if we compared each measurement directly across data centers? In other words, we typically measure flux over time, but what if we measured flux between the two sets of data at the same moment in time? This turned out to be feasible, if a bit tricky.
Out of 144 measurements, the two data centers were out of sync 140 times (97%). As we data scientists like to say: Yikes! The average temperature for those mismatched measurements was 138°F, also higher than the 10-minute flux measurements. Keep in mind that these measurements were nearly simultaneous (within 1 second, generally) and that the results were delocalized and depersonalized. Typically, “new xbox” isn’t a heavily local query to begin with. So, this appears to be almost entirely a byproduct of the data center itself (not its location).
So, What Does It All Mean?
We can’t conclusively prove if something is in a black box, but I feel comfortable saying that Google isn’t simply injecting noise into the system every time we run a query. The large variations across the three keywords suggest that it’s the inherent nature of the queries themselves that matter. Google isn’t moving the target so much as the entire world is moving around the target.
The data center question is much more difficult. It’s possible that the two data centers were just a few minutes out of sync, but there’s no clear evidence of that in the data (there are significant differences across hours). So, I’m left to conclude two things – the large amount of flux we see is a byproduct of both the nature of the keywords and the data centers. Worse yet, it’s not just a matter of the data centers being static but different – they’re all changing constantly within their own universe of data.
The broader lesson is clear – don’t over-interpret one change in one ranking over one time period. Change is the norm, and may indicate nothing at all about your success. We have to look at consistent patterns of change over time, especially across broad sets of keywords and secondary indicators (like organic traffic). Rankings are still important, but they live in a world that is constantly in motion, and none of us can afford to stand still.
Maybe we're all just part of one big split test?
LOL
(i wouldnt be surprised at all)
SERPception! (InSERPtion?)
Press the button on this page for a noise that acts as an appropriate accompaniment for the above comment
Whether Google is messing with us or not, they're messing with us.
No ranking tools out there are going to pull the kind of granular data you pulled for this experiment.
We can buy up an assortment of proxies and run delocalized/depersonalized rank checks across multiple data centers multiple times a day, trying to arrive at the Truth for a given keyword position, but the point here is there is no one big Truth. There's nuance and flux across the board.
We're going to look at rankings so long as we can pull them on a semi-automated basis. They still help us get a snapshot, perhaps vague, of the competitive landscape. But this is not a 5-by-5 signal by any stretch.
My takeaway from this experiment: place even less emphasis on rankings as a KPI (and they were already way, way down the list).
Nice work (as always), Dr. Pete!
Yeah, in the sense that rankings are being calculated on-the-fly and are highly personalized, localized, etc., the algorithm is messing with us. It's just the nature of the beast in an increasingly real-time world. Every observation collapses the wave function.
I don't think that rankings are without value, but we have to look at the very big picture, especially over time. Obsessively watching them is like day-trading - 0.1% of people doing it might get rich, but the other 99.9% will go broke and crazy.
Some of the possible reasons could be the insertion of News Articles, "Xbox" related terms should be generating sufficient amount of new results. Could not this also be a factor that if you were hitting multiple datacentres for such queries, then there could be an obvious flux?
This actually brings up two very valid points that make this a challenging analysis. I'm guessing that Mozcast is looking at Google Organic results and not factoring in any vertical results such as image, video, news, etc. Is that a correct assumption?
But my belief is that the amount of change in SERPS is probably very directly correlated to things like estimated search volume, cpc, and number of new pieces of content published daily. Chances are (especially this time of year) that there are exponentially more documents published daily which Google has to analyze relating to "new xbox." And perhaps the flux is caused more by the sheer volume of pages.
The second issue is the fact that Mozcast is looking at change in URLs, correct? Not change in subdomain or root domain? So if one of those news sites or tech authority sites publishes updated information, the SERP changes, but "not really."
From my own data collection I see that the flux occurs mostly "below the fold" meaning that most of the change is coming from sites that are moving within the positions 4 to 10.
Love the study, and we've been doing similar things. The good thing these studies help with is convincing stakeholders that SEO is not a "one time project" that you complete and then move on. It's something that needs constant attention, testing, and reporting. So I appreciate all the tests you do.
We do look at URL-based flux, yes. I'm exploring sub-domain flux, to see how much it varies. You're correct that the Top 10 for "new xbox" tends to have the same domains, but sometimes new content comes into play. In many cases, though, rankings changed without new content.
I mentioned QDF, but didn't really dive deep into it - yes, I think the high flux query definitely has more new content in play, more new links, more new social signals, etc. The data paints a picture of a highly dynamic, real-time environment, where Google is almost constantly re-evaluating these signals.
It's not shocking. We know that queries are essentially real-time. There's no master table where Google stores your ranking - that's just not possible, especially since a surprising percentage of queries are ones they've never seen. Rankings are generated at the moment you search, which means any new information that's available will come into play. So, the question is, essentially, how often does Google re-check that information? I think the answer is: "Constantly".
Dr.Pete you are absolutely right,every quires brings new content and discussion also because every person have different thoughts. And yes there is no master table available and no masters in the whole world who will ranked forever on front page of Google.
Our keywords rankings change every week (according to SEOMOZ reports) . One week it is 40 position up, the other week it is 25 positions down. This is not for all keywords. Some keywords are always consistent. I think the keywords you rank for has some kind of relevancy attribute that is special for our websites. (or pages). The more relevant a keyword is to your content, the more consistent it is. This is most definitely more complex with several factors but this is just one.
" I think the keywords you rank for has some kind of relevancy attribute that is special for our websites. (or pages). The more relevant a keyword is to your content, the more consistent it is." This makes a lot of sense really. No wonder new sites experience spikes in ranking after a period of time. The content itself becomes centered more or less on one specific subject, hence the keywords enjoy higher rankings. Google most likely makes predictions based on the entire content that exists on a site. Predicting user intentions in a way. Put in links from relevant sites into the equation and the flux is probably even lower.
Hey Dr Pete. As you know, I have been looking at this question pretty intently over the last few weeks. I recently wrote about using a weighted moving average of rankings as a proxy of competitiveness or closeness in actual rank of two URLs in a SERP, which doesn't necessarily fall on fixed integers as rankings do.
Did you happen to see the location of volatility in the SERP. Did fluctuations appear to occur more near the top or the bottom of the SERP? Were these unique results or simply different from the previous search? ie: maybe there were only 20 variations. Were there any multi-position changes? Did the #1 ever become unseated?
I think it is highly probably that Google has a margin-of-error with its rankings calculations and, when that MOE intercepts the MOE+-Rank Score of those above and below, Google uses some amount of randomization if only to collect user statistics to help validate one ordering over another.
Saw your post and we discussed the methodology internally here - definitely an interesting idea.
I have location data and actual flux levels, but the granular analysis didn't look terribly useful for this particular post (and you can just keep digging, I'm finding, to the point of insanity). I will say that "temperatures" for "new xbox" did vary quite a lot - there were some small shuffles and some large jumps. It wasn't any simple pattern of back-and-forth or a big series of one position moves. It was a fairly active landscape.
The trick with the randomization is that I'd expect to see more of it in low-flux keywords. I really believe this is a combination of new data coming into play and a secondary effect of a highly-distributed data system. The more often Google is processing new information, the more data centers are going to be slightly out of sync. I'm not sure they need randomness - it's inherent to the system.
Is this just personalization? (even though its turned off) or panda playing with positioning to get clicks? When scraping over and over from the same ip and not clicking could google be showing you different results to try and get a click?
Over time, it seems that our de-personalization methods have been pretty effective. If this was a result of some kind of historical data or location data, I'd expect to see a more consistent pattern of flux across all three keywords. Could Google be personalizing differently depending on volume or some other attribute? Possibly, but nothing in the URLs really cries out personalization.
Great post, thanks for the insight. I'm a strong conspiracy theorist myself and believe that Google purposely shuffles rankings to throw us off from time to time. I also believe that post penguin it just takes a lot longer to get sites ranked up. Good to see some data that shows just how much the game changes daily.
Brilliant post Pete!
I have a theory (well, first, a NON-theory). The root of this can't be on-page or link metrics, simply because of the time it takes to recrawl & index, and the enormous amount of time it takes to recalc link metrics.
OK, on to my theory: Google is tracking click behavior real-time, and using it real-time (ish) in the algo.
Maybe even (for performance and architecture reasons), the click data is being captured and used separately in each data center.
Hmmmm.....
So, here's where I think it gets tricky. Yes, crawling and re-indexing takes time, but it's happening all the time. I don't think it's a cyclical thing (like once/day) since Caffeine - I think Google is always incrementally rebuilding the index in a distributed fashion. So, some process that started a day ago might finish, and then a process that started 1 day - 10 minutes ago might finish. The processes each took a day, but the changes occurred 10 minutes apart, if that makes sense.
I was thinking the same thing Michael. We know that in Adwords, Quality Score is calculated every time a search query triggers an ad, and one of the component factors of Quality Score is click data. The "real time" parallels seem quite striking.
Thanks for the great post Pete, some absolutely fascinating insights.
Didn't Matt Cutts' announce just recently that Google had over 500 updates to the algorithim last year?
This averages to about two updates per day. Yes, this does not take until account the results changing every ten minutes but we tend to agree that the freshness of the content does play a factor.It's no longer about being an online and having a website, you have have an integrated marketing mix if you are hoping to survive in today's world.
Great insights as always Dr. Pete! And, now I have to choose a different blog post topic...lol - Although, I think your post is better than mine would have been on this subject. Several days ago I decided I would try something that Fabio Riccota suggested when he spoke at MozCon last Summer. He recommended targeting keywords ranking on the 2nd page of the SERPs for improvement because he theorized that they might be easier to improve than words higher up or futher down. So, I pulled all the keywords I have ranking on Page 2 for one of my campaigns. Our of curiosity, I ran them in Rank Checker, because it had been 3-4 days since SEOMoz had last updated the rankings. I about fell out of my chair. Not a single one of them was on Page 2 any more, Some of them weren't even in the top 20 pages of the SERPs. Instead of panicking, I decided to walk away and check again in the morning. The results were interesting, now a few of them were back on page 2, but most were on pages 4 and 5. I waited a few hours and ran them again. This time about half of them were back on page 2, some had moved to page 1 and still many were on pages 4-5.
What you say about tracking trends over time, especially with groups of similar keywords seems to be the way to go. If I spent my time trying to chase around keywords displaying this kind of volatility I'd never get any productive SEO work done.
Thanks for yet another awesome post!
A lesson in why traffic is the key metric, not Google search ranking. Do your due diligence on search and keep developing other traffic sources. An added bonus is that a considerable amount of this effort will be beneficial in search rankings (social media, blogs, etc.)
I think it's been said above but part of this is that if you are 10th and you update that fancy shmancy blog, you could easily move up (or down) and affect rankings. People *are* affecting their rankings every second of everyday so it would continuously shift. I agree with Dr. Pete above in that rankings don't "change" now, they simply move all the time as new content is added to the net.
This is one reason we've been shifting away from a real focus on keywords to a focus on traffic and eventually to revenue. Who cares if you rank 11th today if you're 4th tomorrow and 22nd on Saturday? It's unpredictable because of the constant change.
Keep up the good work Dr Pete.
I'd really like to see this study done across a much (MUCH) wider array of keywords, with attention paid to words (or sites or niches) that might float around a lot due to new content versus those that might not.
I'm willing to guess that there are certain niches of words or phrases that move around more than others, and as there are so many categories of words as there are words, the data set is so huge, we may never be able to see the clusters and patterns without an extremely massive analytic process.
Yet, we all know some words that haven't budged for years, and some that seem to have slid radically, so some sort of computer-assisted crowd-sourced way of thinking about it (and respecting privacy) might be a way to do it.
Maybe something on the scale of Seti @ Home or something?
Dr. Pete as you mentioned before the query "new xbox" deserves freshness and it almost dynamical - agreed! But 2 others are enigma...
There is a view that commercial and informational types of queries have different SERP algos. Also you should hold some parallel with "popularity" of these queries. For example (Adwords, USA, English language):[new xbox] world:40 500 local:22 200[blood pressure chart] world:110 000 local:74 000[fun games for girls] -But in this case here is no any relation between amount of queries/month or just not enough data to insect it deeper...
Anyway thanks for your analysis.
Great study Pete.
The fact that the 2 datacenters were out of sync 97% of the time is quite shocking but maybe worth repeating this a few more times?
In a seprarate note, I'm not sure if the findings can 100% rule out the random injection model concept.
I should elaborate and say that I think it rules out a simple random injection model. In other words, I strongly believe that Google is not just adding a +/- every time you run a query. If they were, we'd expect a baseline flux across all three keywords and maybe even with every new search for the same keyword.
That doesn't rule out a more complex model, but the big question that comes to mind for me is: Why? What does Google gain? Just causing us trouble isn't their primary goal, and anything they do to add randomness also wreaks havoc on their ability to measure and collect meaningful data. The algorithm is already so complex that any later they add can really make a mess.
Great post, you do an excellent job of isolating variables and really applying scientific methods to your research, it's great stuff.
On the idea of perhaps why Google would randomly change results, i had thought i read something in regards to 'over-optimization' and maybe taking a look at the sites whose results for keywords went down and sort of taking a look to see if there were overly anxious SEO's possibly 'chasing the algorithm' so to speak. So instead of making site adjustments based on user needs, making adjustments based on algorithm changes, ranking changes instead. Do you think Google looks at this data and uses it at all?
What might be happening is that G is just trying to handle load. So while each data center is tuned for the geographic area it is serving, say California or Canada, when there are spikes in loads, the nearest data center with spare capacity takes over - and naturally you will get different SERPS, because that data center has been calibrated for a different area.
I agree with larry.kim (and others) above about natural noise.
This experiment is far too simple and uncontrolled to even come close to even hinting at a possibility that Google is or is not injecting noise. The assumption that random queries will see consistent flux is a bit ridiculous, considering the difference in noise you'd expect anyway due to:
Those are just some basic factors to consider and each has much more granular breakdowns (changing ranking signals is obviously in the hundreds if not thousands) and then combinations of those. I'm sure the engine is crawling fast and iteratively. In the Adsense example, Google would be able to easily justify crawling an Adsense page every time it runs from a financial perspective. Anyway, this is something you can test.
Just take query type and user feedback. Google isn't collecting too much useful data by returning the same SERP over and over when they have a large set to experiment with (competitiveness). Add to that, they can consider what they know about a searcher (even your bot has location, maybe behavior, etc) it makes sense for diversity do increase in certain situations. Like during certain times of year (say a few weeks before Christmas) a transactional query (like "new xbox") has multiple user intents and has god knows how many pages vying for a spot on page 1 with heavily SEO'd efforts, traditional marketing, no marketing, black hat SEO, e-commerce SEO, blog posts, Q&A's, etc. ad infinitum (practically).
MAYBE you could get enough information to claim your premise that Google might be randomly injecting noise IF you:
I think we're all better off running with the conclusion that there is enough natural noise going on in Google's search algorithm now that artificial noise is unnecessary for them to inject. I think this gives SEOs too much credit. More importantly, we should realize that reverse engineering is not going to happen without very big teams that are MORE focused than Google's own search teams...and probably using machine learning to pick out any patterns and iterate experiments on those.
Essentially I think you need to build a search engine / site that is capable of building pages how you expect Google (and users) want them. The app must have amazing abilities to build complex pages, measure their performance against the search engines, learn, iterate, etc. I've had some ideas & discussions about this, but it really only makes sense to do it as part of a business, where the site profits off of it's improved success. It requires smart teams comfortable with big ideas, lots of data, who can clearly understand the limits the feedback data gives them. Anyone who wants to do that with me, should know I'm interested. :) That's serious technical SEO, and I see that as the only way to draw any meaningful conclusions from experiments on the algorithm.
Again, I'm only claiming that this is evidence against a simple model of random noise - one where Google flips queries by some small amount every time you check them. I'm not trying to prove that they don't add any noise at all to the system - just make some observations that start to address the questions.
I feel like you're arguing for an all-or-none approach, and in my experience that ends 99% of the time in none. I don't think we have to be perfect at this point - I designed MozCast as a starting point to being making observations about what's inside the black box that can help validate (or dispute) what Google tells us. I have no desire to reverse-engineer the algorithm - only to observe, correct bad information when possible, and hopefully, on occasion, come away with something actionable to marketers.
If that means I occasionally put out data as a straw man - I'm ok with that. I try to be transparent about the data, the methods, and the limitations. Attack it, suggest alternatives, and absolutely (please) do it better. I'd love that.
True, but if you don't try to reverse engineer the algorithm or at least intensely try to do something close to that, it seems like you'll never get scientific information, only anecdotal. I am suggesting a big idea, and I'd love to do it to, which probably means I should. It's not quite my job right now, but I think that's no excuse too.
Random noise just doesn't make sense to me, although targeted noise does, as well as algorithmic complexity (which appears to be noise). Anyway, my suggestion besides the big idea approach, is to start by filtering out the likely and/or known noise as much as possible. Seems like there were too many variables in the beginning of the experiment and too many possibilities around random flux. Someone suggested that maybe flux is injected once a week or only on Tuesdays. Consistently random noise seems to be what you're looking for, but that of course couldn't be labeled random, unless you ruled out all other possible intended effects (knew the algorithm). Perhaps I'm just not clear on what possible conclusions could be drawn from any given outcome here.
Anyway, glad you're running these types of tests.
Does anyone beside Googlr actually know what is really going on ?
Back in the days (early 2000), it was easy to observe Google DC shuffling SERPS every night.Why would this change in 2012 ?I don't get the point of this post. There is nothing new or terrible here.
Great datamining! How about social and vertical searches? They also impacts rankings, and changes very, very often.
Looking at the phrase; "new xbox", it could show a Youtube video (e.g. review) or talk about a new xbox. Even blogs could easily have fast content on this subject.
It's just Google split testing the internet, looking at bounce rates and time on site.
Brilliant study Dr.Pete. I was just wondering over this ranking fluctuations for the past few weeks. But can we simply ignore these fluctuations? because as the keywords are shifting to 4th page or 5th page it can affect the organic traffic to our site also right? I have seen a drop in the organic traffic to my site in Google Analytics recently and is it due to this SERP flucuations? Is there anything we can do not to undergo these ranking fluctuations?
Nice write up but the keywords goes up and comes down because of the searching habits of the visitors, it also indicates that with the passage of time it keeps changing according to the trends and visitors searching habits.
it's depend on google updates and google split testing I agree.
Wow, not only the post was great, but the discussion it generated also! I love reading interesting studies like these, they always teach me new stuff.
Hi Dr. Peter,
This is a nice experiment. I think you’re the only SEO who did this. A lot of SEO are making a lot of guesses on how Google treat these keywords. Even I also thought that Google is messing the SEPRs to sabotage SEO industry. You shed light on this matter. Thanks for that helpful information.
Thanks for this. I've been watching my ranking for one of my keywords fluctuate every day for weeks now. Although I'm getting concerned now because I'm slowly moving lower and lower in the top 10. The change isn't drastic. Its been about 1 position every two weeks. But because of the noise some days I'm down 2-3 and other days I'm even or up 1 position.
Hi,
great to know! I was wondering why my ranking was fluctuating that much intraday and day to day for specific keywords. I'm a newbie at SEO...
Thank you
Wow, I`m late to the party but I just found this great thread. I just want to add something that might add to the discussion (or not!). G wants to maximize is ROI. It`s obvious but it seems easily forgotten in most of the discussion. So if I was in charge of G, I'll want to give more customers to companies who pay me. One way to do this will be trough personal search history. So the closer a web user get to the buying phase, the farther away website with forms who don't pay me will go. So my paying customers will make more money. Of course it's a gross oversimplification of what I'm trying to say. But you get the point. Multiply this by all the factors they know about the sites, about you, about trends in the marketplace, about the competition, about how many clicks on average it`ll take before you buy, about localisation, about everything in fact and then it makes sense that all this information in real time could change the result as you've experienced. This could explain in part the change, of course there's a zillion other factors in play. The funny thing is nobody can really find out about this. There's too many data to analyse in close to real time and nobody can do it. We'll know for sure when somebody from G will confirm or deny this.
Great post Dr. Pete. I've always wondered about keyword fluctuations but I've never had the resources to perform such an experiment. I love it when other SEOs do experiments like this!
Keeping keyword fluctuations in mind is extremely relevant, whether you're using it as part of your daily SEO work or in present it as evidence of SEO success. It seems to me that it's better to err on the conservative side when taking credit for keyword ranking improvements... actually, I consider rankings as secondary to organic traffic, CTR and conversions as KPIs. But, I suppose too that it depends on just how hot and cold specific queries or industry terms run.
i feel the answer lies in the keyword traffic. "new xbox" have 201000 borad search followed by "blood pressure chart" and no search at all for the keyword "fun games for girls". check google
keyword tool. You can always observe that very long tail and less searched keyword serp won't change often.
Great post, Dr. Pete! I'd love to see more follow-up, like a look at the broader implications. How do these fluctuations affect the user? Do regular people (non-SEOs) find it frustrating if searches they perform frequently return varied results every time? Or do they even notice?
I have one site and don't have content on it. My keywords are regularly dancing. How can I control Keyword dancing and get them into good position.
Awesome experiment! Liked your analogy about the world moving around the target.
Ahah yeah, thought the same.
Great post indeed.
What I think would be really interesting would be a comparison to other search engines. I realize the majority of us focus on Google, but is this level of flux "normal" in the search engine industry as a whole, or is this unique to Google?
Great experiment. This helps everyone lay to rest any Google conspiracy theories.
Thought provoking and yet another reminder that it's well worth spending the time to understand the SERPS for you target keywords, noting their make up and their volatility before you start optimising/creating/promoting content.
Be nice if the rank tracker also reported the SERP temperature over time too!
Good article. At some times I do feel like Google is just sitting back laughing at me every time I check rank as they sit behind the scenes changing the results.
I would like to see more testing on this... just to see if Google doesn't revolve around the earth, the earth revolves around Google :-)
Here is an interesting article from Bill Slawski discussing additional unexpected flux in the SERPs
https://www.seobythesea.com/2012/08/google-rank-modifying-spammers-patent/
If such factor and pattern really exist (or other similar ones) then it's another chase around the maypole.This of course applies on younger sites and over longer periods of time.
The keywords with a high search rate tend to fluctuate more. Very interesting to watch google insights to see these changes throughout a week, month, and even over the years.
I have been checking a bunch of keywords everyday for last 40days. No big changes are found, but some times it creates misunderstanding between my clients and me. Again, I have to educate them about the all algo changes and what are you talking about :)
Anyways, sometimes it seems that Google is also messing up in serp on the reporting days of my projects... lol... it puts me in trouble... :)
I agree with @Gamer07 somehow but i also believe that it is also the matter of crawling. If your website has much better crawling rate than your competitors then there are chances that keywords will stay on their positions for the longer time. Normally, i have observed that when you leave to build links for a page or keyword then after sometimes it goes down because its crawling has gone down.
I completely agree to the moderator of having a dominance in terms of position changes which is directly proportional to the crawl of your website, as some websites do tend to get crawled more often then others such as news, rss, etc.
This is something I believe all of us SEOs have been thinking about. I have never had the time to even try and think of a way to test these ideas. Thank you for taking the time to "go down the rabbit hole" that is Google rankings. It's greatly appreciated Dr. Pete!
Client: Why did specific ranking change by x this month?
SEO: Read this: https://www.seomoz.org/blog/a-week-in-the-life-of-3-keywords
"...don’t over-interpret one change in one ranking over one time period. Change is the norm, and may indicate nothing at all about your success. ...look at consistent patterns of change over time, especially across broad sets of keywords and secondary indicators (like organic traffic)"
I love the insights you always give to the SEO community. The real question here is what does this mean for rank-tracking software across the board, even yours. I also think this adds a lot of questions for local SEO as well, because it gets even harder to get info on local results because they are location-based by nature, and it's impossible to have servers everywhere.
I hope this gets explored moreZach
Great post DR Pete - just a quick question on the data. You said:
the Top 10 SERPs were crawled and recorded
Do you mean the top 10 Search Engine Results Pages (100 URLS's) or do you mean the top 10 Results (10 URL's)? I presume it's the latter...
Top 10 URLs, yes (first page).
FYI - I posted a brief follow-up on Google+. I tracked the flux across two data centers for the low-flux keyword ("fun games for girls"). Should've done that originally.
I think it does rule out a random injection model, but I think it also doesn't give any real indication of why specific keywords go up or down, which could be down to a number of factors. Tweaks to the Google algorithm will make a difference at different speeds, because the cycle of Google's spidering will be all over the place. It doesn't re-index all pages for a given keyword at the same time, so it is constantly re-ordering things as it indexes new things and re-indexes old at varying rates. Also, Google has its SEO anti-spam model, which may be at work deranking pages temporarily that are being changed in order to determine SEO spam activity. So, rank changes could be temporary because of on page fiddling and they could also be because of what is happening on other sites in the index around your pages.
I see that you did this all depersonalized and delocalized.
Do you think the variances are greater or lesser for someone who is logged in and being served localized results, aka the normal searcher?
Thanks for posting this, its really interesting. I often see similar results, I think you can check your keywords too regularly, and I find the top 10 results change positions almost hourly.
I like your title, is google just messing with us, perhaps also it can see there isn't a great deal between these sites and is rotating them to give everyone time at the top?
Excuse brevity -on a mobile device here. I think of the noise differently. I think it's more of an artifact of how google is built. That they don't compute finite rankings for an infinite number of different possible searches, but rather, I thought they just do some super quick estimations on the fly. And as a result, because they're just estimating an answer in a stable yet slightly non-deterministic way, you get those 80% serp listings change every day. In wordstream I do similar things -- estimating rather than doing full calculations for complex operations like grouping hundreds of thousands or millions of keywords on the fly. Rather than analyzing all of the data which could take forever, I just take a sample and make rough estimate, and as a result you get slightly different results every time you run the tool, but it's pretty close and the benefit is that it's +100x faster.
I think that's entirely possible - the computation itself may be such that the result is non-deterministic, especially when coupled with a distributed data set. I don't think that explains the wide variation in flux between keywords, though (at least not entirely). We're seeing a combination, I suspect, of an artifact of how the system itself works combined with the dynamic nature of the keywords themselves and their SERP environment (volume, competitiveness, "freshness", etc.).
sure it does. the "flux" is to be expected in a non-deterministic system where there is no official ranking computed for every possible search. i'd expect the flux that you're measuring, would vary based on the values being used to compute the estimate, which varies by keyword. the question you should be asking to prove this one way or another is not to look at the number of changes for a keyword over time, but rather the magnitude of the change, and if the sampling of rankings for a particular keyword forms a normal distribution around the average ranking value (which I would expect). So again, i just think that the noise at baseline, which varies by keyword, is just noise that has to do with how the system is built to provide instantaneous results.
I think if you were to find statistically significant changes in average rankings, then there is probably something else at play. like Google messing with SEO's, or changes in the values that are being used to calculate estimates for a particular search (eg: different link counts, or new documents, etc.) or algorithm changes, or SERP ranking a/b testing, personalization, etc.
I read that as basically saying that flux is caused by: (1) the nature of Google's system, and (2) the nature of the keywords themselves. That's exactly what I'm arguing. I don't think you can say that the computation "varies by keyword" but then separate out the characteristics of that keyword or the sites that rank for it. The actual changing nature of the data in the "wild" has to have an impact - the question is just how much.
yes, that's what i was saying. that if you get changes other than the expected noise, it has to do with other factors like different link counts or new documents or changes in the other 200 signals - changes to "data in the wild" as you put it.
in summary, my main point was just to point out that while your research has uncovered that 80% of listings change in a week, and that certain keywords change more frequently more than others - that i believe that most often (but not always) that this is just an artifact related to how the Google system was designed in the first place - that in order to compute instantaneous search results, big shortcuts must be taken, which results in the flux in rankings that you are reporting.
Hi Dr Pete ,
Good post . We are also following serp's 6 to 8 keywords for a clients website from past 40 days , we have seen a slight shift in positions as you wisely said these might be related to query volume & other external factors as they keep varying from day to day .