Experiment I – 4 Keywords, 5 Methods
In Experiment I, I took 4 of my client's most sought after keyphrases (from a popular 1-word query to a long-tail 4-word query) and measured rankings for a week using 5 different methods:- Default – Standard, logged-in query
- PWS=0 – Adding the &pws=0 query parameter
- Logged out – Standard query, but logged out of Google
- Rank Tracker – Data from SEOmoz's Rank Tracker tool
- GWT – Data from Google Webmaster Tools
Practically speaking, rankings for this particular set of keywords didn't vary much across methods. Keyword 1 tends to bounce between the #1 and #2 spot, which the Logged out ranking showed, and there was some disagreement about Keyword 3, but the differences were mild at best. All methods correlated strongly with the default search (r = 0.97 to r = 1.00).
Experiment II – 20 Keywords, 6 Methods
Of course, this was one set of data and only 4 keywords/phrases, so I figured I should up the ante. I pulled the Top 20 search queries (by impressions) from Google Webmaster Tools and did a second round of one-day measurements. I also added a 6th method, "Caribou". No, it's not a secret codename – I took the laptop to Caribou Coffee to pull a new IP and tried a logged out search from there. Experiment II's numbers turned out a bit more interesting:
This one takes a bit of explaining. Graphing 20 keywords x 6 methods is ugly at best, so Figure II shows the number of times each method's ranking varied from default across 5 levels, from ±0 (same ranking) to ±4 spots. The PWS=0 and Logged out groups showed the least variation from default searches, with the Rank Tracker, GWT, and Caribou groups showing more variation (especially at ranking differences of 1-2 spots). Correlations ranged from a perfect 1.00 for the PWS=0 group down to r = 0.71 for Caribou and r = 0.69 for Rank Tracker.
What Does This Mean, Exactly?
I'm glad you asked. Of course, this doesn't mean that the Rank Tracker and Caribou measures are unreliable. On the contrary, both correlated strongly (r = 0.90) with Google Webmaster Tools (GWT) rankings. What it does suggest is that, given enough keywords, rankings do vary a fair amount depending on how you measure them. There are a couple of general conclusions I think we can draw:1. PWS=0 Is Unreliable
I'd welcome more data on the subject, but it looks like the "pws=0" parameter does little or nothing for many queries. If you're using the de-personalization parameter and taking the result at face value, I'd strongly suggest you reconsider. It does appear that turning off personalization may affect some geo-targeted personalization, but the query parameter doesn't make Google completely ignore your search history.
2. "Logged out" Probably Isn't
Google's announcement last year as much as admitted this – if you think being logged out will de-personalize your searches, think again. The open question is: just how much different is it? This data suggests that being logged out has very little impact on rankings, assuming that you're on the same machine with the same IP. Move to a new machine/IP, and the difference is much more substantial.
3. Second Opinions Are Gold
There really is no gold standard. The rankings in Google Webmaster Tools are the closest we can get to being inside Fort Knox, but these numbers are completely opaque and many SEOs have reported occasional rankings that differ wildly from observed searches. If you rely on rankings as a primary metric, get a second opinion, preferably either a fully logged-out ranking on a browser/IP with no history, or by using a 3rd-party tool like SEOmoz's Rank Tracker.
4. Skepticism Is Healthy
It always makes sense to check your facts, and search rankings are no different. Rankings vary – you can occasionally type the same query twice in a row and see two different results. Smart SEOs have already diversified, considering metrics like search traffic and conversions. On the other hand, even across these test cases, rankings don't vary a huge amount. So, don't panic, but as always, the key is not to put too much trust in any single number.
Very good analysis. And yes like some people said it definitely be nice to go more indepth on this research with my case studies and experiments.
Its funny cos I realised a few days ago that logging out doesn't do jack for personalized search too! These days the main things I use to get a better idea of ranking is Google Webmaster Tools and a seperate browser which I use strictly for checking ranks. With that browser, I always delete the history and don't use it for anything else. I find this works best for me, at least till something better comes along.
Cheers
Between you and the rest of the world I get the feeling we should stop paying so much attention to tracking rankings...
Perhaps measure success instead by metrics like traffic, conversions, and sales...
But while were on the subject of rankings, what about testing various browsers? Chrome provides a wonderful little magic trick where you can open an "incognito" window that allegedly doesn't leave traces on your machine. The same goes for the wonderful private browsing function in Firefox.
As always, excellent work Dr. Pete. Looking forward to our next appointment.
Although limited in scope it was very interesting. I think the point about hitting different data centres is very important too.
In practical terms when you're reporting, I find the easiest thing to do is pick one method and stick with it so at least you are consistent. Then make it clear to your client what you're doing and why rankings are not the same for everyone. At least if you report in the same way each time you can usually get a reasonable idea if the sites you work on are going up or down in the SERPs. Simple things I'm sure we all do!
Agreed about making it clear to the client Bludge. Although it's not the easiest point to get across when the client sees on their own browser that their results are different than you report.
Thanks Pete, great study. I'm not sure which measure you used, but I want to share something with you and other SEO researchers.
When conducting a correlational analysis with ranking data, make sure to use Spearman's or Kendall tau rank order coefficient. Pearson's coefficient, the most commonly used, should not be applied to rankings.
It's a small point, but important for people in our industry to know.
Thanks again.
That's certainly a valid point. For small-scale analyses like these, I find the difference to rarely amount to much, and I always prefer to do the quick-and-dirty approach first (just to eyeball the data). For larger-scale studies, though, ranking data is definitely a bit different beast.
Dr. Pete, That was a very interesting post, thank you. It's interesting to hear that the ranking don't vary a huge amount. Should be interesting to keep track of how this changes in the future. Thanks again. By the way, are you a real doctor?
-Matt, Tiepedia.com
Word on the street is that his Dr. refers to the moniker "Dr. of Love". So don't pay attention to anything he says about psychology.
It worries me how often I get this question. Are there that many fake doctors running around? Does it shock and amaze people to think that I could have an advanced degree? ;)
As the main character in Mumford once said, "No, I'm the fake kind," which is to say that I have a PhD in Psychology (cognitive/experimental) and am not an MD. By the way, Mumford is a very cool movie that was totally different than the previews (it's NOT a romantic comedy).
We had a bet in the office it was actually Dentist Pete - doesn't have the same ring though.
Great post - I'm based in Spain and all my clients are UK based or Ireland. The combination of personlised search and difference in data centres is making ranking reporting very difficult. This brings us back to multiple KPI's and encouraging the monitoring of traffic as much as possible.
When I was in acupuncture, we had a joke designation for MD's: RD = "Real Doctor", since clearly obviously of course, no other medical professional really is real or knows anything at all. ;-)
Logging out of Google Account does not disable personalized search completely.
As a matter of fact you have to 'Disable customizations based on search activity' after being logged out too. The boring part, is that you have to disable this from 'Web History' regularly cause it automatically turns on all the time.
But I doubt a regular Google user will know about this feature in the first place. Personalized search is practically enforced on the majority.
With something like personalised search results, it's difficult to take what you're saying into account without some further testing. Performing this amount of tests doesn't really give us a viable source that we can refer to and use in any way. It would be fantastic if you could increase the scope of this operating in both timeframe and amount.
I'm probably going to get boo'd for saying this but I honestly did enjoy your post and have taken on board what you've said. I just can't take what you've said as something viable until there's been some more testing.
That being said, thank you for taking the time to write this; it's much appreciated.
I completely agree that this is just the tip of the iceberg (and thanks for the very diplomatic comment), and I don't want people to walk away from a small-scale experiment with wild ideas or thinking I'm making outlandish claims. If there's a take-away, it's that I think people should be aware that rankings are dynamic, and measurement does matter. You shouldn't take those numbers you see on Google every day or week at face value.
Of course, I think good SEOs already know that, but a little reminder never hurts. I would, of course, love to see follow-up experiments (and am pondering that myself).
Thanks - I was worried about annoying people, so I'm glad you've taken it well. Even though parts of my comment above are negative towards the article it's nothing personal; it's just me taking a step back from being blinded by the numbers and having a think about the consequences of what you've said above.
I'm really really interested in personalised search and can't wait to see what studies you perform in the (hopefully near) future.
Just a small note, if you ever need any help with the studies, just let me know. I'm always available to do some extra curricular work if it means taking the load off your back and all I'd ask for is a mention in the article.
It's sometimes hard not to be defensive as a blogger, I admit. On the one hand, I want to welcome sincere criticism and feedback, so that I can improve. On the other hand, it sometimes annoys me that a thrown-together rant will get 50 "Hell yeah!" comments where a 3-4 hour research project (not counting write-up time) will get torn apart for not being "perfect" or all-inclusive.
Pardon a huge tangent - I've learned to ignore the general negativity, but what bothers me is that the ultra-critical reactions to data-driven posts ultimately discourage more SEOs from trying new things, experimenting, and reporting on their findings. I absolutely think that we have the right to question bad data, push authors to do better, and generally be skeptical. On the other hand, I'd like to see the community support the general idea of publicly-reported research, even if it's small scale or the author is a novice.
That's my long way of saying: "Thank you for your support" :)
Couldn't agree with you more Dr. Pete. Knowing how hard it is (measured in time and sweat equity)to put together a post, it bugs me when people thumb posts down so summarily.
I'd be the first to admit that I'm the type of person to be a defensive person when it comes to research, so I admire your responses to me so far. Glad you took it for how it is and didn't go into a blind rage, find me and rip my balls off (haha).
I wasn't trying to discourage SEOs by my comment, I was rather trying to do the opposite. Encourage people to research and bring out more conclusive results. In no way am I questioning the validity of your data, it's just that coming from a background studying some psychology I always tend to look at the wider picture... In this case, a study involving a wide-scale test would be more appropriate.
Once again, I appreciate everything that you do.
I'd be very willing as well to help with research. This could really be interesting to see the spread of results as I will soon be in the UK and I believe that you, Dr. Pete, are in Seattle. I don't know where you are Traxor but it could be interesting to see.
I'm based near Birmingham in the United Kingdom. I'm not sure that Dr. Pete and his team are willing to take on volunteers. From the looks of thing, everything will be handled in-house. However, if you want to team up and produce an article for YouMOZ, then I'm more than happy to do that with you. All of my contact details are on my profile.
I'm actually in Chicago - work with the Moz team part-time. I hadn't specifically planned on a follow-up, but feel free to DM me through the site and maybe we can concoct something.
Two examples of personalization -
When I search for Link Building I get a notice from Google that the results are modified based on my history. Guess what site jumped from page 2 #11 to page 1 #7? SEOMoz.
When I search for online MBA Google [wrongly IMO] modifies the results based on my IP geo-location. One state listing jumps from #10 to #1. Another jumps from ?? to #4.
You don't mention if the sites you tested even qualify for personalized results. What does it take to get a site qualified for your personal results? 10 clicks from SERPS? 25 clicks? 100?
It definitely requires more digging, but my experiences so far suggest that personalized search is less about quantity and more about the type of search. Geo-targeting, for example, is one incarnation of personalized search, and I think you see local searches affected more than broad searches.
There was an interesting post at OnReact.com recently where the author tried to trigger personalized search using just brute force (typing in a ton of searches to try to bias results). The impact was negligible at best. Again, this was a small-scale study, but I think the upshot is that personalization is a much more complex concept than we generally give it credit for. You can't just type in "SEOmoz" 50 times and see your Google results wildly change.
Thanks a lot for the research, Dr.P
I've tried testing personalised search but rankings somehow didn't change drastically.
Rolling out personalised search can
1. Satisfy google users - people love to see what they want to see, e.g. if I want to buy something online, I want amazon to be on top.
2. Prevent some "SEOer" gaming google algorithm - more and more business realise the importance of ranking on #1, countless cases that some companies dominated SERP by purchasing links and without any penalty. So if someone doesn't like the website on #1, it will get replaced by other websites (slowly?)
Is google getting too personal? That's google's ultimate goal, to satisfy every searcher. Although it creates trouble for business and SEOers, but for Google and majority of google users, I have to say it's a right move.
Personalization of search is definitely an interesting topic, but I have a hard time devoting much time researching it because it’s ultimately something we’re not going to be able to control. In dealing with clients (or bosses if you’re an in-house SEO), it’s obviously important to understand how rankings could be being influenced by personalized search. For example, you may need to explain to your client/boss that when he checks the rankings of his site he is #1 because he’s logged in and always clicks on his result. Similarly, the ranking for a non-geotargeted keyword might be skewed based on location and the site might not be ranking as well across the country as it is in your geographic area.
Even though it’s extremely important to be able to explain these issues, I don’t think that anything we discover about personalized search results will influence overall SEO strategies. Optimizing for local search is always going to be a good idea because people often search with local modifiers such as city names. Doing this will automatically optimize your site for Google users in that geographic area, assuming you’re actually located there.
You can also leverage personalized results by making your search titles and snippets more interesting, and thus more clickable. If people click your results more often, then you will theoretically be influencing personalized results. Of course, we are all trying to do this anyway as part of standard SEO strategy, so again I’m not sure how understanding this changes how we do SEO.
I don’t have much to say regarding the ranking data shown in the article. However, I wanted to throw in one thing I’ve noticed while checking rankings with tools that access ranking data via Google’s AJAX Search API (https://code.google.com/apis/ajaxsearch/). I’m not sure if SEOmoz’ Rank Tracker uses this or not, but the tool I use accesses this API for its data.
We all remember that Google used to tag certain pages as “supplemental results” in their SERPs. They stopped doing that, but in reality the supplemental index still exists. With a modified site command (site:domain.com/*) you can find out which pages on a site are in their main index, which indirectly tells you what pages are supplemental. So how does this tie into rankings? Well, I noticed that the AJAX Search API only seems to return pages that are in Google’s main index. For that reason, if you are running a monitor on a keyword that is very targeted or uncompetitive, then the ranking data might be incorrect. In the real search results, Google will grab some pages from the supplemental index to increase the number of results. The Google AJAX Search API will not do this, so the ranking data you gather from your tool of choice may say that you’re not in the top 10 (it will return your most relevant page in the main index) when in reality a supplemental page from your site is ranking in the top 5. It’s also possible that you may be ranking worse than the tool says if your competitors have supplemental pages that are more relevant than your page in the main index.
What about simly searching in "private mode" with all history/cookies disabled?
This seems to me to be the best method to ensure objective serp results.
I second that, especially since pws=0 seemed to disable geo personalization. This would be whole different way of tracking rankings.
Great post though! :)
A 1 to 2 position change does not have to be attributed to personal search, it could be the difference between 2 datacenters for example. I don't think this experiment has little significant value, interesting subject nonetheless.
Absolutely. In the first experiment, I think the differences were almost entirely due to regular fluctuations. In some ways, though, it makes the uniformity of some of the methods (like pws=0) even more interesting - I honestly would have expected more variability, even in this limited data set.
After a search I always click "View Customizations" then: view results "without these improvements'. Doesn't this address the issue - and shouldn't it have been included in the testing?
The methods list started to get a bit out of hand after a while, but that would certainly be worth looking into. Given Google's recent statements about personalization when you're not logged in, I suspect that this won't fully address the issue. A call to the API (like rank-tracking software uses) is still going to be a less biased form of ranking data.
Great study. Many useful results. Although i think if we have more data we can have more useful and reliable informations.
Google personalization in SERPs = telling me what I already know!
And Btw, Google seems to be getting more personal in last few weeks...
I'm curious how logging out + clearing browser cache + deleting cookies would affect rankings. Would that give true, unpersonalized search results?
I'm all for just sticking with one method, as someone said above, and explaining the differences to clients.
You simply need to logout of Google Account ... >> AND << make a simple search, then you see 'Web History' pop up top right corner on Google GUI. Click on it and you'll see:
"Your search results may be customized using search activity from this computer."
Click on "Disable customizations based on search activity."
and it's fixed TEMPORARY :P
ALSO don't clear cookies, cause it's auto set ON again! Google is just pushing personalized search, very clear.
This post reminds me of all the times that my clients would tell me their website were already at the first page when they were applying for seo when i tried to tell them that actuall they weren't.
Yet they were around 9-10th page. Seems they used to stare at their own websites all the time lol
Interesting post! While the test was small in scope, the points you offered up were reasonable conclusions.
As the engines move toward greater personalization and away from serving consistent results, rankings will only become more difficult to track accurately. And what does a number really tell us about the position of a site on a search results page anyway? With blended search, social results, and any number of beta tests affecting the look of every SERP differently, I'd say not a whole lot.
While a larger test would be interesting to see, I think the takeaways would remain the same. Personalization has made tracking rankings more complex and less accurate, and it's essential to convey that to clients and managers. Posts like this are a real help in doing that!
"Logged out" Probably Isn't
Funny but scary at the same time. I've been waiting for information like this, great test. I'm actually quite pleased to see the results. This definitely confirms that there will be an eventual shift to analytics from ranking reports to measure the results of seo efforts. I think although it may be a bit of a hiccup for seos now, in terms of reporting results, it will help us not lose site of the metric that actually matters - converting traffic.
Now this is what I joined SEOMOZ for. Many thanks for this study, it's really educational. Not just 'on topic', but also as a marker on how to make further studies.
I had guessed that there must be a difference between my machine and my clients, basically because the visits keep rising, but I see less position in my personal searches. At the end of the day I had just reverted to my IP / Log Analysis charts for an accurate view of what's going on. IMHO too many trust WMT's to give them accurate info.
Many Thanks
Col:-)
Are we sure that enforcing personalized search on the Google users majority is a desirable thing? This will lead to a less variety in the users navigation! Am I wrong?
Really the study is appriciable. Good job, but the matter of being looged out from the account and checking the rank details is not known to me. I will surely have a look over it.
Interesting research, i didn't realise they could change so considerably between logged out, and logged in. I'm not sure I like that... does that mean than when I search for something on my computer and (say, something that i search for often) when a friend searches for the same keyword, but doesn't usually, we would get pretty different results?
If you searched for the keyword and clicked on it, yap its gonna be pretty different from what your friend sees when they search for it and don't click on it.
The joys of Google's personalized search.... Even thing is personal!
Interesting subject! If you however want to search with a different IP address you could as well use Opera and then configure the proxy server. This saves a trip to the local coffee place (altough I could see some advantages in that as well :))
Back on subject: It is interesting how search results vary from one query to another. It makes me wonder about the how personal search might interfer or create additional possibilities/ difficulties on seo.
It would be facinating to add to this study on a larger scale - perhaps to even test the variations for localized searches. Nice work, well done.
What about trying searches with a proxy I used abm.me Then searched my website then used Google from my ip (I had to click the view customizations then show without improvements) and got the same results. According to whatismyip.com abm.me is located in AZ Im in Ca.
This experiment needs more data points to provide any useful info IMO.
This is clearly a small-scale study that could be expanded on in many ways, but in this particular case, I really wanted to use client data - specifically, the kind of rankings that each of us track every day (as opposed to generating 100s or 1000s of keywords just to run a larger experiment).
This was partially for my own edification - how much do the rankings I measure each week REALLY vary across methods? Also, I think there's value in putting together small-scale studies that other SEOs can easily replicate. I want to encourage people to experiment a bit on their own, and not always be overwhelmed by data sets and analyses they don't have access to. Anyone with a browser and Microsoft Excel could do what I did in this post.
Hey Dr. Pete. My short answer after the Google Buzz debacle would be "yes", they are getting too personal. :)
Thanks for sharing the data you found re: the personalized results that Google has been feeding us more of as of late.
I never did have statistics in college, so I don't really understand the mechanics of what the detractors above are pointing out.
But whether or not it's statistically correct, it's data that I can use to bolster my discussions with clients. So thanks for sharing.
How come all these SEO experts don't know that you simply have to disable "Enable customizations based on search activity." from "Web History" when logged out of Google Account to see actual rankings?
lol or I'm missing something here ...
Google has more or less stated that even that's not enough. There's still some level of personalization tied to your IP, even if you're "fully" logged out (essentially, there's no such thing as of December). I think that trend is only going to continue - Google will personalize, geo-target, etc. and will give you less and less control over it. The danger, as this progresses, is in assuming that your search experience matches everyone else's.
With all due respect, geo-targeting, etc is something else and I don't think it was the main issue here.
Following my instructions you will see non-personalized results and don't need any fancy code or use particular tools. Obviously location of IP does effect serps but that's same for everyone else in my own country at least.
Sorry, I'm using "personalization" in the broad sense of any time Google alters your results based on specific, personal information (including your location), and especially if it's done without your knowledge or when you think you've turned these features off. The core point is to help people understand the need to perform a sanity check on rankings, no matter what their chosen method is.
Pete - Very timely post. Thanks!
Your method of checking rankings is similar to one I started using last month. I use:
1) PWS=0
2) Google Webmaster Tools
3) SEOmoz RankTracker tool.
I combine the three sets of rankings and divide by three to get an average.
I considered using Google results with personalization, but I quickly saw that my company's site ranked much higher that way, which made me think the results were too skewed to trust.
I also tried using SE Rank Checker (a Firefox add-on that is part of another SEO toolbar). That list, however, was almost always lower in rankings and sometimes wildly different (+/- double digits at times). I threw those results out, also, writing the tool off as not very accurate.
I considered the logged out of Google stats, but everything I've seen and heard seems to indicate those were no different than logged in stats, so I blew that off.
I think I'll add in Pete's idea of checking with an off-site PC from a different IP. My home laptop should suffice since I don't do work searches on it.
I'll admit my method my have holes in it. However, when the boss wants rankings and is aware of personalization skewing them, I had to come up with something.
I echo the comments on different Google datacenters reporting different results. Amongst my collegues in my office, we're seeing rankings vary from #1 to #3 to #7 for one term with personalization on, and sometimes I'm seeing that myself in two different browsers on my PC within one minute of testing these.
My experience with my method, albeit very limited so far, is that all three of these ranking tools are within a pretty narrow tolerance level of each other (typcially +/- 2 to 3 ranking positions, with many exact same ranings for at least two of the methods).
I both hate and love the idea of personal search.
It has great potential to easily become either an anvantageous feature, or a major hindrance for businesses fighting for good SERP rankings.
My feeling are mixed, but if it ultimately turns out to provide the most relevant results to the user, I'm all for it.
We shall see.
I think results would tend to favor listing companies and directories, because users would be more likely to have clicked those in the past than individual company websites.
But it really depends on the industry.
I really dislike personalized results.
I only discovered this the other day.
A site I've only just started working on, and ranked around the 40 mark, all of a sudden jumped to number 2 for the keywords.
But it only did that every other search (must have been hitting a different server).
Now I use private browsing in Firefox to stop any cookies that might give inflated result to regular searches.