LDA is remarkably well correlated to SERPs, but by substantially less than I thought or claimed. Expected correlation (as measured by expected spearman's correlation coefficient over our dataset) is 0.17 instead of 0.32. I found a mistake with the calculation that produced the 0.32 score.
0.17 is a fine number, but it is awkward having previously claimed it was 0.32.
Some Implications:
Statements I made in the past two weeks along the lines of "LDA is more important (as we measure it, yada yada) than other ways we've found to evaluate page content, and even more surprising than any single link metric like the number of linking root domains" are incorrect. A corrected statement would be "LDA is better correlated (yada yada) than other ways to measure page content relevance to a query that we've looked at, but less correlated (yada) than several ways to count links."
Topic modeling is still another promising piece of the pie, but the slice is not as large as I thought. Or claimed.
Slightly long winded description of the bug and what evidence there was of it:
I was looking into the discrepancy between Russ Jones's chart, which showed roughly a linear relationship between SERP ranking and sum LDA scores, and Sean Ferguson's chart, which showed a huge jump for the mean LDA score but the rest pretty random. Russ Jones had based his chart off our tool. Sean based his chart off the spreadsheet. After looking at it for a little bit, it was pretty clear the source of the discrepancy was that the tool and the spreadsheet are inconsistent.
I tried reproducing a few results of the queries in the spreadsheet using the tool. After about a dozen, it was clear the spreadsheet (compared to the tool) had a consistently higher scores for the first result, and consistently lower scores for the other results. That is technically referred to as the ah shit moment.
I reviewed the code that differs for the web page and the spreadsheet, and found a bug that explains this. When generating scores for the spreadsheet, it caused the topics for the query to be largely replaced with topics for the first result. This made the first result to be scored too highly, and later results to be scored lower.
Excluding the first result from every SERP, the bug actually made the results less correlated in the spreadsheet, but the help getting the first result correct was enough to boost the correlation up a lot.
A Few Related Thoughts:
- When I release statistics in the future, I will continue to try to ensure we provide enough data to verify (or in this case show a flaw with) the calculation. Although I found the bug, it was only a matter of time before someone else would try reproducing a few of the queries in the tool and see the discrepancy. So releasing data is a good way to ensure mistakes get discovered.
- The actual expected correlation coefficient, 0.17, still is, at least to us at SEOmoz, exciting. But the smaller number is less exciting, and it really really sucks I first reported the expected value for the coefficient as 0.32.
- Some have claimed there is something invalid with measuring correlation by reporting the expected value of Spearman's correlation coefficients for SERPs. They are still wrong. Two wrongs don't make a right. My programming mistake doesn't invalidate any of the arguments I've made about the math behind the methodology.
- Mistakes suck. I feel shitty about it. I'm particularly sorry to anyone who got really excited about 0.32.
Here is a corrected spreadsheet and below is a corrected chart. For historical purposes, I'll leave the incorrect spreadsheet available. I'll edit the two prior LDA blog posts to include links to this one.
[Update around 7PM: I'd really like this to over, but maybe it is not. Sean pointed out to me the mean for the second result is higher than the first result. I don't have a good explanation for why they would be. Hell, the few I looked at the spreadsheet today the first result is higher in the tool and the spreadsheet. I was rushing to get a correction out there - it may well be I fucked up the correction, and maybe in a way I could have noticed exactly the same way. I'll update this post when I know more, but I think 0.17 really might not be the last word. I may not have it in me to do a mea culpa post for the mea culpa post, but I'll update this post with whatever I learn. Seriously sorry. Just treat all of this as suspect until we know more.]
If there are in fact 200 factors, then each individual factor is going to be a tiny fraction of that. I think people get too lost in any single factor as being the golden ring of SEO. People really need to grasp the visual here. Topics are a factor. That's something we should all be able to accept as reality. Whether it's 0.1 or 0.3 or whatever, is not something ANY of us has the capacity to determine. Trying to do so leads to arguments and rebuttal rants beyond insanity. Really.
And since the LDA tool is not based on multi-word phrases, which Rand already acknowledged, people really need to get over trying to think LDA or the LDA tool are that golden ring. And instead, should just focus on across the board best practices, not getting hung up over and over on any single factor.
heh, I made the same statement in reply to Rand above.. cute :-)
@OlivierAmar LOL well that just shows that 1), I don't always read all the comments when I have a burning need to add my 2 cents and 2) my comment proves you're a genius :-)
Amen!
I really feel like the only harm that may have been caused by the "hype' would be to a new SEO who hasn't learned yet not over-react or under-react to anything.
Clark,
I think that's a valid concern - people new to our industry, however it also relates to many people who have been around a long time and don't stay focused on the bigger picture.
My view on the data as I originally stated in the first article's comments is essentially that I've perosnally seen topic model importance in my own work, and that LDA seems to be a valid representation of my own work. Yet I also personally don't ever get caught up in hype, hyperbole or marketing spin even if it's subtle or unintended. I'm not most people though. Most people get totally lost in that, in their search for that golden ring.
I do think Rand does a reasonable job of making disclaimers with these "studies", yet in all honesty, with so much information overload to digest, most people gloss ofer even those efforts.
Is the answer to then not share this info? Maybe. Maybe it's to share this info with the first paragraph being an entire H1 red colored disclaimer.
People also need to recognize, yet most never do, that the data being provided is coming from a company that is in the business of selling its analysis tools. So whether it's intentional or not, the spin will, as a natural aspect of the business process, drive users / visitors to want to make use of the findings. That's not a terrible, evil thing. It's a business reality.
And most people are blind to all of these other considerations.
Thanks for standing up and admitting that one, Ben. It would've been easy enough to just quietly notice the mistake and never let the outside world, so I hope people at least appreciate that aspect.
I want to say the following on a strictly personal level and not as someone who represents SEOmoz:
I think we may have gotten carried away on how we put the LDA data out there, especially because it was during the PRO Seminar and everyone was naturally excited. What I hope people understand, though, is that Ben and Rand (and many of the rest of the team) sincerely love this stuff. That excitement was NOT a rush to be first to get industry attention - they love to do this work, they love to learn, and they love to share that knowledge. The enthusiasm may have gone too far, but it was sincere.
I do think, though, that there was a valid core point in all of this about LDA. It's not that LDA is new or that we somehow are laying claim to it or think we're the first ones to ever study it. It's not that LDA is the end-all and be-all of SEO and everyone should rush out and rewrite their sites (please don't do that). It IS that (even with this correction) we've gone so far toward the link-building side of SEO that we sometimes start to discredit relevance and on-page factors. They still matter, and the upshot of these studies is that they still matter a bit more than we previously thought. I think that's an important conclusion - it shouldn't uproot how we do SEO, but it's a signal that we sometimes go overboard to adopt new directions and discard the old ones too fast.
Completely agree, well said ... it was a little hyped, but not worthless data ... LDA or topic modelling in some form is being used by Google, so sort your context out 'on page' around your key words ... simple.
Agreed, the calculation was erroneous out so the correlation sounded more impressive than it actaully was. The fact is the advice in the previous posts is still solid and a foundation of SEO. I feel for Ben, it must really suck to mess up in front of your peers, but I applaud the way he's managed it in this post. May be a good thing, he'll probaby quadruple check everything before posting it publicly in the future.
Thank you Dr. Pete. After reading the various sets of posts and comments about LDA on seoMoz, this is the first comment from an SEOmoz Staffer/Associate that I have come across in these posts as objective and not defensive while putting it all in perspective.
I am quite surprised at all of the fire (both positive & negative) about this topic. Danny, in the original LDA post, made a great comment that most people seem to overlook (both in blind focus on links as well as the geewhiz response to the LDA findings): "Yeah, it always comes back to content, doesn't it."
From the Google Webmaster Guidelines
In terms of content strategy, this will probably bring a few thumbs down, their guidelines seem pretty straightforward and would naturally imply some sort of topic modeling algorithm being used (whether it is LDA or not). That should not come as some sort of a surprise, but the reactions (both positive and negative) and language used indicate otherwise.
Last night, Gillian presented the SEOMoz research on LDA to the Dallas/Fort Worth Search Engine Marketing Association (https://dfwsem.org). As president I specifically asked her to share the SEOMoz research to our group - and in light of today's announcement (less than 15 hours after presenting it in its orginial form) - I would gladly have her do it again!
SEOMoz is on the fore-front of SEO research and Ben's work is both inspiring and insightful.
First, SEOMoz shared this as "research" and released a LAB tool for testing and evaluation - so they could obtain more input. Never did they say this was the end-all, be-all of SEO but that it had an important and noticable correlation.
Second, repected members of the SEO community were in contact directly with Ben in following up on his research. In fact Gillian provided Ben's email to us last night so we could follow up with him (which our MIT graduate with two search patents appreciated since he had some questions and ideas). At all times SEOMoz and Ben wanted outside input - to either validate, refute or correct their work!
Third, there were a lot of disclaimers around this work which again shows that this was very preliminary.
Fourth, mistakes happen - which is why SEOMoz and Ben opened themselves to public questions when they made the announcement.
Fifth, the blog site was the perfect venue for this and their initial excitement was well justified, even if in-correct.
Sixth, Topic Modeling is a part of a bigger puzzle and now that it has been identified as part of this puzzle, it can be further reviewed in context of other modeling.
Finally, on a personal note, I hope that SEOMoz will continue its ground breaking work and willing to share both the good and the bad, regardless of the temporary reaction to this correction. Too many "experts" do not posses the caliber of character and intellect of SEOMoz's team and we in the SEO community should support and help our fellow members like SEOMoz and Ben who strive everyday to make our industry better!
Wow, just read this and wanted to send a thanks your way. I really appreciate you taking the time to write this. A lot of what you touched on in here we talk about A LOT internally, and we are continuously pushing ourselves to research, analyze, consider, dialogue, etc. While this may have been the huge discovery we had origionally thought, I personally am very proud to work at SEOmoz for many of the reasons you listed above. :) I am also -- time and time again -- impressed by the calibur of our community members, not often do you come across comment exchanges and conversations with such time invested in them and such great insights. So thank YOU for that.
I think the problem here is not hiring a statistician - if I am correct Ben is exactly that. The problem is that Ben is human and humans, as we know from daily experience, are flawed.
In academia one would usually enter into a system of peer review to ensure that mistakes such as this do not happen - this is how journals maintain the credibility of their articles. Perhaps it would be a good idea to share results and data with two people outside the SEOMoz offfice before findings are published, along with large enough data sets that results can be reproduced? This would help to eliminate errors just like this one.
I have experienced the wrath of the community myself in a statistics-related post before, I wrote + instead of - in notes I made making things rather wrong, meaning that I will be adopting peer review before any more "hardcore" posts are made because that wrath was entirely justified.
Oh, and I still think that LDA correlation with rankings is an important result. It was never a game changer, Matt Cutts et al have hinted strongly about topic modelling for long enough now that if we aren't thinking about it we aren't doing our jobs properly, but it does prove statistically that on-page content and context are important enough to think long and hard about them.
This is still a very exciting result and insane respect to you Ben for being so open in this post. We've all had those ah shit moments and some come back to bite us harder than others but it doesn't really change anything at the end of the day - this is still exciting research and still shows some very interesting things. Keep up all the good work.
Hey Ben, I really admire the way you've handled this. Jumping out in front of the bus of cranky SEOmoz bashers the second you found out you were wrong and making a post about it was even over the top for TAGFEE.
A note for any "anti-SEOmoz/Ben/Rand" individuals out there. I invite y'all to show even a fraction of the class that Ben and the gang at SEOmoz have in tempering your commentary.
Here's a fact that has 100% correlation - People make mistakes. This isn't the space shuttle we're talkin' here, it's a new SEO toy to play with. It costs everyone nothing but the time it takes to play with it.
Regardless of whether it has a ,32, .16 or point whatever correlation, it's up to the individual to test it before leaning on it. Did it work for you? Great. You're ahead of the game. Did it return nonsense results for you? Great. Now you know you can ignore it.
But to continue spending your time lambasting SEOmoz for releasing a free tool to try is pointless. And to use the line "They are industry leaders, they have a responsibility...blah, blah, blah" is treating the rest of the SEO community like a bunch of little children. Adults take responsibility for their own actions.
If an adult follows bad free advice blindly then they're responsible for that choice they made. If the advice they follow cost money, then they can either ask for a refund or just leave the paid service. But either way, it's up to them to take responsibility for themselves.[/rant]
I have to agree. I have sat here debating whether or not to post my comment because it's clear that these 2 or 3 people who are mad at SEOmoz for this are trolling the comments and thumbing everyone down. So I didn't want to get the thumbs down! Oh well...
I'll admit, I got excited too, thinking that improving the topic relevancy (topic modeling) of my page would rocket it straight up to #1. So I reviewed the homepage content of one of my sites, edited it to improve topic modeling. Waited. I did nothing else. In a few days that site went from #11 to #4 AND #5 (an indented result, hooray). So was I disappointed? Yeah, a little because it didn't go all the way to #1. But was I so ticked off that I decided to lambast SEOmoz for informing me incorrectly? Of course not. Who cares about the hype. Even though they were not 100% correct, they weren't 180 degrees in the other direction. LDA helps...just not as much as they thought. This testing just confirmed another of the 200 ranking factors. By the way, this page has a PR0.
I hear 'ya about the thinking twice before jumping into this LDA brouhaha. I almost didn't comment as the last time I jumped in between Danny and Rand I had a hoard of Danny fans (I count myself one of them by the way) give me a massive thumbs down response.
And it looks like the down thumbers are back today [sigh]
Don't give in to the thumbs!
Fear not the thumbs or the "tumms" as just about every kid where I grew up would pronounce it.
Now I'm curious, where did this happen to you? The recent WBF? Regardless, you're conscientious, engaging and a real community builder. you'll have to try a lot harder (or be more like me) if you want to piss people off.
I'm still dumbfounded why people would have the audacity to complain about some hot new information, given to them, for completely free (for betterment of the industry as a whole)
I don't know what is wrong with the world we live in - but I for one thank SEOmoz for everything they continue to contribute to the industry.
I must return to read all of the comments, but a few things remain REMARKABLY (and game-changing-like-ly) clear. ;)
SEOmoz must be terribly influential for such belly aching. Which means, I'm so happy to be gaining friends and peer relationships through SEOmoz. I'm also looking into other worthwhile communities, of course, but "Yay, for me!"
... but ...
"But yer mom said 'reverse engineering the SERPs' and 'remember 45 mins ago' on twitter."
"You used the word 'remarkable' and 'surprising correlation' so the rest of the world will bend to your will."
"I'm going kvetch because you're a profit company."
"My pants are too tight, why?"
I was at the event and smiled at the tweets and so forth. Why? My thoughts and my ability for critical thinking are my own. For the rest of us ... we're grown-ups. Stop the belly aching! The reasonable course of action is to point out that time and tons more alpha-beta (yes I said it) testing will eventually show us the validity of the tool. Talk to any tool developer and they will tell some interesting stories about successes and failures of cool ideas. Ah, I hear that thought in your head ... read on.
Let's not confuse malicious (or ignorant intent) with SEOmoz's TAGFEE. Transparency means that we talk about stuff. In this case we're talking about stuff early on in a product life cycle. We're getting real time insights into the wins and misses of this tool. I find it very exciting.
If the tool offers any benefit to my clients, then I'll use it. Time will tell, not Twitter or the liberal use of adjectives. But we knew that because we're adults with critical thinking abilities.
Yeah, spotted it right off.
Actually, Ben, I'm just in awe that you were able to come up with a cohesive formula to explain something we could see but not understand, then test that formula to the extent as to notice an inconsistency to your findings that to be honest most of us would sum up as "okay, I might be off a tad."
Then because of who you are and how you think you announced it to those of us that rely on your work but don't fully grasp the amount of data you're using to postulate these numbers. (speaking strictly for myself since I know the rest of you are erudite)
I thank you for your work and diligence to detail that enables such a tool. I'm glad you're on our side.
Hi Ben,
That must have been a bad feeling when you first found out about your mistakes. The good thing is that mistakes lead to better results in the feature, so that your future researches will be even better :)
The fact that you (and the rest of SEOmoz) is so transparent about mistakes is such a good thing and a good eye opener for me, although the fact remains that it sucks that it happened. Also the speed in which you discovered your mistake is good enough, with the topic still burning in everyone's ears, so no one is on a absolute wrong track.
Thanks for the clarification and the new data! I do agree that it is still amazing (or exciting as you call it) that this is of such high value in the total SEO mix...
Yeah - the correlation difference should mean only that using this incarnation of LDA isn't as predictive of rankings. The tool's usefulness and effectiveness for improving rankings is something we didn't know at launch and still don't - it will be interesting to see how SEOs use that and what impact they derive. Anecdotally, we've heard both very positive and "no change" stories, but definitely not making any claims on that front yet.
Perhaps even more important than "it works or not", are the total lack of accounts about how LDA killed a site. It's a work in processes with potential. I'm excited about a tool that offers insights. Interpreting those insights, estimating value and then acting upon them is still our job.
Ben,
Thumbs up to you for coming on here and being honest, as someone else said you could have just quietly carried on and people may have been none the wiser. For being honest you have gained a lot of respect from me and a lot of other moz users.
To quote someone else.. 'its not the **** who makes the mistake, the **** is the one who doesn't learn from it'
Nick
Ha Nick,. that is certainly a quote worth remembering :)
It shows great character to admit your mistakes. For that Ben deserves an applause.
Absolutely right.
Ben, good on 'ya for being open, honest, and professional about making a mistake. The way you handled this was exemplary.
So my takeaway from this correction is that the tool itself is still pretty darn spiffy for gauging content relevancy, independently of the correlation figures. If good content is good and we know and have always known this, why wouldn't you try to use it to your advantage? The new number does feel more correct - I think one of the reasons we were all flabbergasted by .32 was because it was much bigger than we all had felt previously. The research is still good, even if it's now reinforcing what we already knew intuitively. That is in no way a failure.
Oh, and TAGFEE FTW! ;-)
Yes TAGFEE FTW! :)
Your experience with this is like giving an important speech to a large crowd and then discovering you had spilled gravy down the front of yourself. AAurrgh!
SEOMOZ:
I think the answers are:
The difference I see about SEOmoz is that you show the results, explain your conclusions and ask for feedback asap the experiment just after the experiment had been done. And:
Finally, I really appreciate this post, as I appreciated the previous ones (and all the others about your experiments).
Science is experimenting and experimenting is an iterative process based on previous erroneous experiments... and many times from mistakes great things are discovered (just remember penicillin).
But what I appreciate the most is that you, Rand, Ben and all SEOmoz, are fiercely keen on give a scientific base to what would be simply "experience"... coming back to the penicillin example, that was the same: penicillin was used because of good experience since middle age, but was Fleming who told us scientifically what it was and why was good for medicine.
PD: I agree with the comment of Alan here above. Mozzers... don't look to close too the process of Ranking when actually working, or you will just see one factor and get obsessed by it, losing the right perspective... and finally making a bad SEO job. And this is just a practical tip I've learnt from experience.
I don't see the problem. This is still a very good result and it makes perfect sense. It got me hyped but that hype made me think about on-page stuff that is still very important. SEOmoz maybe overhyped this but humans work there and it's prefectly natural to be excited. I might not agree with SEOmoz in every single thing but I like you guys the way you are.
Its always going to be tough doing science so openly in a business setting.
For me this worked exactly as it should. Yes, it wasnt exactly correct, but the process for catching errors was in place. Publishing early is fine (business win) as long as you provide the data for others to validate (science win).
Congratulations on your full frontal mistake Ben! I am serious. That doesn't happen to people who play it safe, never push boundaries or don't innovate. This is worthy of a Seth Godin Award or some other cheery recognition. Keep jumping your motorcycle over razor encrusted school buses, dude!
That last sentence is something worthy of ending my work day on. I'm closing up the computer with that sentiment in mind :)
Great comment! You have to dare to make mistakes if you want to achieve great things.
So, final word(s) on this? I have 2 lessons:
1. Anyone who really believed the hype should be spending more time optimising their clients' sites and not referring to SEOMoz for every last piece of advice they give.
2. People shouldn't be criticising SEOMoz for the open-ness of their studies. Peer appraisals, sense-checking etc are all valid points to add to the discussion but SEOMoz is here to help stimulate debate and should be pushing people to go and find out things for themselves, it is not supposed to be some back door into the Google algorithm. The 'hype' around their announcements is a byproduct of how successful they have been in sharing info from the studies they have undertaken. 99% of the SEO community appreciate this open-ness and very few of us share it in return (I know I don't :)). In short, and in a very 'American' sentence, the world of SEO would be a darker place without them.
As you were.
There was always to likely to be an element of schadenfreude when a mistake like this is made. I went to the SEOmoz Pro seminars in London 2009 and Ben was a really likeable guy, and a crowd pleaser. I think even then people could see that while the current offerings were limited, but the opportunities available for the future are promising. Yes a mistake was made, and yes it was too hyped but I hope this doesn't hinder aspiration to produce further statistical studies at SEOmoz.
I'm just glad that when I make a mistake, it's only a limited handful of people that will pick up on it!
Thanks for the update.
Holding your hands up and saying "whoopsie" takes some balls - good work.
My 2nd son was born around the time you discussed LDA on the blog, so I must admit I missed your explanations about this algorithm (And it seems I'm really too lazy... or maybe I'm just missing too much sleep... to try to understand it now), but I got a question folk: Do I need to follow a course in advanced Maths to be a better SEO? :)Edit: I forgot... Congrats for standing up!
Ben, this is awesome transparency and disclosure. Don't beat yourself up, as others said. It's still really valuable research, and we all appreciate it. Nicely done!
Ben,
Still great work. Very informative...don't worry about the mistake, could've happened to anybody
@Ben: dude, don't be so hard on yourself! You did a great metric exploration / discovery job and the theory validation is just still a bit buggy, so what? The Wright Brothers probably had a few rough landings as well when they had their Big Idea! ;-)
@Rand: you know I sympathize with you. There's a Dutch saying called "High trees catch lots of wind" (Figuratively: The more important the person, as a consequence, the more criticism they get). That's tough some times and regarding your 'blood pressure' comment above: I sympathize with you even more. However, I feel - although you probably wanted to give BH credits for his discovery - that you shouldn't have put him on stage (WBF, clearly feeling uncomfortable, dance comments...), especially without knowing 100% sure that facts were facts. Also this blog post shouldn't be outed at his (sole) expense either.
It's been a tough week for a lot of people. Glad to know I'm not alone in the ability to make mistakes. Thanks for having the character and integrity to be open and transparent about it, instead of trying to spin it or blame it on someone else. We're all learning, and no one expects perfection all the time... Good post.
Ben, don't worry about it, you did the right thing - you made a mistake, you found it, you copped to it and informed everyone of the correction, no harm done. It's still very interesting work.
Hi Ben,
thank you for your post - I do appreciate it! We all know that mistakes can happen and there are bigger problems in the world than a LDA correlation 0.17 instead of 0.32 ;-). But I do understand very well that you feel kind of unhappy....
Ben - I agree with this sentiment: don't beat yourself up. Most us have made much greater mistakes.
Admitting it in a blog post is more TAGFEE than most of us would have the guts to do. :)
Slight maths errors aside, a slight positive correlation for just a single metric is still interesting. This is still one more concept that we should consider, and it's another tool to add to the SEO's armory.
The whole LDA discussion has definitely persuaded me to think more about on-site targeting in the last few weeks - rather than just the blunt tactics of targeting a couple of keyphrases in the key parts of a page.
The SEO implications of LDA concepts have never been discussed and analysed in public in so much detail before; SEOmoz's research is quite clearly a work in progress, so please do keep us up to date on any more info you collect.
Rob - to be fair, I think calling the correlation number "slight" may exaggerate things (or, rather, under-play). While it's not the 0.32 number, here's some comparisons from previous data in correlation analysis work:
Certainly, looking at these, you can see why we were shocked by the 0.32 figure, but the new 0.16 doesn't take away massively from that, it just suggests that A) our model is likely much more primitive than what Google's using B) topic modeling as a whole might be smaller than we though as an algo chunk, though likely still significant and C) we're really lucky to have good community folks helping us catch us when we slip up.
Also thought Russ' work that Ben pointed to above was very compelling in showing that value shifts across the keyword demand curve. Would love to examine that more.
Rand,
We know that there are 200 or so factors by which the Google Algorithm ranks our sites.
That being said, wouldn't all 200 of those show a positive correlation depending if it were a legitimate item like titles, quality/relevant content, page speed etc..? I don't think any item will ever score as high as .6 .5 .4 or maybe even .3. They're all contributing factors that we knew about and know can positively influence rankings. I don't think we'll ever see a negative correlation on relevant content..
These studies feel to me sort of like the search ranking factors where you have most of the industry experts weigh in on. Except that there we get insight as they give their opinions and experience as well. Here we get a kind of a confirmation that I'm not sure was really needed.
I've played with your LDA tool. It gave me a 98% score. I'm still ranked 9th for one of the most competitive terms around ;-)
Olivier
Right right! So, given that something we think of as being very critical to SEO - using your keywords in the title element of a page, has a very low correlation compared to this, it's an interesting additional data point.
With regards to the LDA tool showing 98% on your page - that's great, but just as having the highest PR or the most links or the best keyword usage won't always win the day, neither will topic modeling. It's another piece to add to the competitive analysis puzzle. My best advice on usage is still to do something like this for your keyword phrase and SERPs. Sometimes, LDA might be able to help show why you're less competitive than you think you should be based on links/kw usage/etc.
yes, even though any one factor may never score higher than .3, it's important to know which ones are .1 and which are .17, right? I like these kinds of tests because it certainly helps in prioritizing projects.
So unless I'm mistaken you're saying LDA correlates to rankings slightly less than Google Toolbar PR?
And yet if I were to say "want to know if you'll out rank someone? Who has the higher toolbar PR?" I'd be laughed out of town.
I'm worried you may not be understanding what the correlation data is good for. By the same logic, one would be "laughed out of town" for recommending use of keywords on the page, in the title, in the URL string, having static URLs and dozens of other common best practices.
As several folks have noted, if there are 200+ factors in Google's algorithm, then finding more of these, particularly any that are more interesting/useful/higher in value than the classic ones we already focus on (namely, links and keyword usage) seems wise. Substantive correlation numbers (particularly with high confidence of non-zero correlation, which we can get through the standard error number) may indicate value in pursuing these factors, though correlation isn't casusation (maybe people are just more likely to link to documents that use topic modeling - though we can look at these independently when we do the rank modeling work).
As we were careful to point out when the original post was released and again in the video - this is interesting and unique and may be useful for doing SEO, but at this point, it's just research and a free tool that might help.
It is quite possible I'm not following what correlation data is useful for.
However, I thought in your responses to Danny in one of your previous posts, you stated the correlation of certain things (keywords in the title tag) are so low because everyone's doing it. As a result, the correlation of such factors is quite low.
But for the same logic to apply to LDA, wouldn't most sites need a very good LDA score?
Side note: this is the first (and possibly only) time I've ever regretted sleeping through my stats class.
If everyone is doing something, that makes it difficult to obtain _any_ information about whether it's a factor. It might be extremely important, but you wouldn't know that from a correlation study. It might also be of low importance. You would need to conduct controlled studies to tell the difference.
With so many factors in play, any significant correlation is... well, significant. There are other possible reasons why a factor that matters would still have a low but non-zero correlation, for instance it might be eclipsed by other, more significant factors that have greater variation (link juice being the obvious candidate in this case). Doesn't mean it doesn't matter or you shouldn't try to improve along that axis.
TBPR won't tell you who will rank, but you still want to get links, right?
I agree, the problem with correlation in this case is that you are trying to correlate one factor with over 200. Its like trying to prove that tax policy determines economic prosperity... we know its part of it, but there are too many factors to get a clear answer. Rand's examples above fall between .11 and .2 correlation. I'm no stats wizard, but a 20% correlation to me doesn't prove anything to me. Doesn't that mean at best 1 in 5 correlates? and 4 out of 5 don't? I think this is always going to be an uphill battle, because there are sooooo many factors, that you are bound to get a lot of false positives. Just as keywords and links, I think topicality is an important element of the Google Algo. I have used the LDA tool to find some good topically relevant words (that I overlooked in the past) that have already earned my clients' some new long-tail traffic. I think as it grows it may be even more useful for identifying what i've missed in content creation.
That being said... I really enjoy the SEOmoz research, as a newcomer you have all (including the commenters) have pushed me to learn so much in the past year. We all make mistakes, but in a sense your mistakes have forced me to look at the subject more critically and I have learned much more as a result. for that, you earn a thumbs up.
Hi Inbounded -
Glad to have you with us. Good thoughts.
Re"I have used the LDA tool to find some good topically relevant words (that I overlooked in the past) that have already earned my clients' some new long-tail traffic."
I've had a number of folks report that they have used the tool successfully for the same - finding overlooked topically relevant keywords.
Ben H,
Massive thanks and appreciation for your work, dedication and openness!
Congratulations on admitting mistakes.
17% doesn't sound like a large number though. And the accuracy is mixed together with the magnitude.
Can you print just the coefficient if you regressed them? ("beta") I think that's the interesting number.
@Ben H,
RE: the labs tool, I would appreciate your comments on the effect of hidden text and whether this was considered in the original LDA research. I have run a few tests using the LDA tool in SEOmoz Labs and it looks like it pulls all of the text from a page regardless of its CSS treatment (for example it is including the kw-rich text from hidden divs).
Do you plan on controlling for this in future enhancements of the tool?
If Google is dampening the effect of hidden text, taking that into account would likely improve your correlations, no?
Thanks.
Interesting Blog with plenty to take in. My head is spinning a bit, I think I need to lay down after reading this :)
You know you have a healthy community of opinions when the comments get so deep the column is 1 inch (or 2.54 centimeters if you are from outside the US.
What a discussion.
I am definitely big fan of sharing your studies and findings. I don't have time time do some freaky scientific experiments myself so it is great that you do it and share.
Regarding the mistake, it is great that you admit it and correct the results. I am sure you are not the first or last person to do mistake. Even better that the outcome is quite similar and it proves that LDA has significant impact on SEO.
lol... we're fine with just as long as you don't go work for Google
Do you guys remember cold fusion? Now THAT is how to do hyped!Serious though... props for being truthful bearing all this.
What about exact match domains or domains that have the exact match as a large part of the domain? That alone would skew LDA rankings for top ten Serps.
So if I recall correctly, this is the 2nd of three high profile studies that SEOmoz has published & promoted heavily, and then had to go back and say WOOPS! We're actually wrong.
While I appreciate you guys admitting to the mistake, I would like to know what you as a company plan on doing about this in the future.
Do you plan on hiring a statistician and releasing your studies for peer review BEFORE broadcasting incorrect information and hyping it as an SEO game changer?
Hi Ben - always good to catch you :-)
We certainly plan to do lots more like these. As you know, we release lots of studies, research and information about SEO all the time. I'd say our track record is reasonably good, and we love to release early and get feedback from the community to help us find errors like this one (it was actually through reviewing work others had done that Ben caught our slip this time).
We may, in the future, leverage outside help, though we've done that in the past as well (though not formally). I feel that our obligation is to continue publishing new and interesting materials, being transparent about our imperfections and correcting when we find errors or additional nuance. I find that position far preferrable to being scared into not publishing or not researching.
Rand,
While I don't neccesarily think you should refrain from publishing, I think you most definitely should refrain from the kind of hype that always seems to accompany these studies.
Members of your own staff called this an SEO game changer and said that Ben had reverse engineered Google's algo. Combine that kind of hype with having to back track like (in much the same way you did when proclaiming NoFollow still worked) damages your credibility.
It's become a pattern of behavior that is disturbingly predictable at this point.
You are dramatically overstating the case. Getting excited and tweeting a line or two during a conference session does not warrant the level of criticism you've assigned and repeated. If you do believe the 2-3 tweets you're referring to are egregious enough to solicit your level of response, you must take equal issue with that same level of "hype" seen on Techmeme/Twitter/Techcrunch/SEO sites/etc about dozens of products, companies and launches every day, yet it seems uniquely targeted at this organization.
It's getting such that I'm worried your motivations have less to do with substance and more to do with personal conflict or politics. I don't believe anyone in our profession is helped by that.
Rand, putting something in print (whether a tweet or not) is still putting in print. You say 2-3 tweets, I pulled 6 just in a few minutes of writing my blog post. You pulled back the hype significantly in your post and whiteboard video, but you still used words like "remarkable" etc.
You say I'm over stating it, but the last paragraph of this post makes me look freaking psychic.
It may be I'm not giving you enough credit for other studies you've released like this, I remember the NoFollow still works, a study about Link correlation to rankings, and this one.
I absolutely do hone in on your company specifically because a) SEOmoz has one of the highest profiles in the industry b) you have a history of this kind of thing.
While you may think it was just a few excited tweets, it was enough to get a large chunck of the industry buzzing about it, and as evidenced in the comments on your previous posts, people were making decisions based on this incorrect conclusion.
We've had this conversation before, but you're an industry leader and with that comes a HUGE level of responsibility. If Blueglass, Raven, SEObook or SearchEngineLand had the same track record and put out the same kind of info, I'd be hammering them just as hard.
Unfortunately, I just don't believe you. My opinion is that you are playing to a classic web archetype - one that's brought you attention, traffic and enjoyment at the expense of others. I don't believe we've got further to go in this conversation other than to agree that we strongly disagree.
SEOmoz will continue to do work like this. We stand behind this work (0.17 correlation with rankings is still, in every sense of the word "remarkable"). We love positive, constructive criticism (for example, you could have pointed out this error, which would certainly have been more valuable than your "hype" remarks) and will continue to leverage the SEO community (who's been so generous) to review our data, research, theories and posts.
If you are not amenable to our style, I'm afraid that's tough. Just as I do to those who don't believe SEO is a real practice or that all SEOs do is sell snake oil, I'll simply invite you not to participate. It will likely be beneficial to both our blood pressures.
Rand, 3 years later and you are still using the troll defense with people who disagree with your tactics? How's that working for you?
You were quoted, by your mom, as saying on stage, "Remember, like 45 minutes ago, when links were really important?" Your mother also insinuated in a tweet than Ben had reverse engineered the Google algo with this study... yet people who call you out for overhyping it are trolls? Seriously? That's where you want to go with this?
For real now, that is what you are saying? That other people can't be trusted if they disagree with you, and that if they do then they must have ulterior motives?
That's really the story you are going to stick with on this?
Michael - You weren't at the event, but those who were know those comments were tongue-in-cheek. I have found that engaging with you in the past has been unproductive, so I'll leave it there.
It may be true that those at the event knew the statements were "tongue-in-cheek", but they were tweeted publicly by a member of the SEOmoz team. It loses its tongue-in-cheekiness in 140 chars or less.
Hey, I think the point is that everyone was amazed when this came out. Hell, it is still pretty shocking. SEOMoz didn't run out and put ads on TV or write press releases, or run adwords campaign, or do anything aside from a few tweets and then a blog post a few days later. I don't think that is "over-hyping". Admittedly, though, it is a difference of opinion.
OK, first off, bringing someone's mother into an argument, even if she's relevant, is just low. Second, you really can't tell those were jokes?
freejung, she's not "someone's mother", she's the president of the company. Nice try though.
Of course she is, but when you linked to her tweet, you didn't use the anchor text "your company President," you used "your mom." That's a deliberately provocative word choice and you know it.
Edit: I have to say, flaming someone on their own blog for admitting a mistake and posting a retraction of their incorrect data (or, if you insist, flaming someone for getting excited because they thought they had some interesting data and then found out later that it wasn't quite as interesting as they thought) is a pretty sad and sorry tactic IMO. It implies things about one's character that one might not wish to imply in public. Just sayin...
And I imagine if they shared anywhere near as much statistical information and data to support their research so willingly (and so regularly) they'd be in the same position here.
SEOmoz provides these insights for free and has always strongly promoted the way of thinking that an SEO should test any conclusions before looking at it as a panacea to SEO win. Perhaps more importantly, I sincerely doubt how many people understand or care enough about correlation and Spearman and whomever else to "assume" just because SEOmoz said this is important I'm going to change my entire strategy- I know I don't.
The people that went through the trouble to peer review this work deserve a tip of the hat. Meanwhile, it just goes to illustrate that SEOs should be testing these hypotheses for themselves (and probably not on client sites) rather than assuming any one source is going to get it right every single time.
I would much rather see the other folks you mention step up and publish more, put themselves out there for critique, and take one on the chin when they find a mistake, than suggest that SEOmoz stop doing it.
I think you're missing the point of what Ben is saying and saying correctly.
He gives props to Rand and SEOmoz for the work and the transparency they do. What he's asking is that they verify their data, with an outside external source, before they go out and hype it everywhere as "reverse engineering" and a "game changer".
SEOmoz is one of the two highest profile companies we have in our industry and by far the most open one. That being said, and just like Ben said, with that kind of exposure comes responsibility. There are enough haters out there that love these kinds of mistakes. Moreover, I get enough of a headache when my clients and junior staff (who I make read SEOmoz through our premium account) run to me with plans on changing our nofollow and content strategy based on faulty reports.
I'm sure those same people that tear down Bens work would love nothing more than to be implicated in the verification process and give their feedback before. What would happen would be less mistakes (I said less, not no) and better studies.
I see your point, the only issue I still find is that in our industry if you sit on something and don't share it, something will inevitably change. If you look at any "blackhat" forum or site almost anything on there that people are willing to share no longer works.
I would rather see SEOmoz writing about things that still work before they've got the official correlation down pat than play their cards closer to their chest. If you want openess, sometimes you're going to get it wrong. I agree that it wouldn't hurt to have someone else check over the information/work but the risk there is that by the time you get someone in to check over something you've been working on for a long period of time it just increases the chance that the "news" is no longer relevant.
Don't think SEOmoz has any intent to share misinformation and I can't blame them at all for getting "hyped" up about their findings. Genuine mistakes happen and genuine acknowledgment of said mistakes are the right way to deal with them.
If no one else wants SEOmoz' info until it's passed loads of external review fine by me- Rand can you just shoot me all future findings in an email, I'd be more than happy to verify whether they work for you :)
Olivier - you make good points - not all of our information is correct nor is it all up-to-date (particularly old guides/blog posts). That's a failure on our part and one we are working hard to correct (tough to balance the "go fix something old" vs. "go build something new" of course). However, on the issue of overhyping LDA specifically, I think we've been very careful to note how early in the process this is, how we suspect this isn't precisely the model Google uses, how we don't know what value SEOs can/will extract and how our research may have holes/errors (and we've invited the community to help find them).
We intend to keep publishing in this manner (and, as I noted in a reply above, may leverage some outside professionals as we've done in the past). Hopefully the caveats and careful wording from our posts and formal announcements will help assuage the concerns about hype, but to be fair - this is, at least to me, really exciting stuff. That passion will probably naturally seep through a bit, but my hope is that it's preferrable to the alternative.
Hi Rand,
Don't worry about negative comments. My suggestion is to involve some pro members before releasing anything. It will be of good help to avoid any roll backs.
Regards,
Preet
Personally, I think "hype" is a non-factor here. When SEOMoz makes a finding that could potentially make a major impact on how SEO should be done I want them to scream it from the rooftops. The issue in this case was that a mistake was made.
By posting the findings in a very public manner they essentially did submit the study to peer review and any errors made were quickly identified and addressed. Kudos to SEOMoz for sticking their neck out and sharing their knowledge and their process. Their williness to be so open is what has earned them their excellent reputation. Trying to tear them down for a lack of perfection should earn a reputation of another kind.
Perhaps these comment sections should be reformatted so people's replys arent put into a four word width column?
You know, hype can't exist without the audience latching on to it. All of us in the audience are the hype machine. Maybe we need to take a second and reflect how we can add value now, and keep moving forward.
"Leavin' Haters Like Las Vegas, In a State of NV..."
Outside help is available if you need it Rand. All you have to do is ask. ;-)
Heavily promoted? Hah. I heard about the study 2 days before they even put it on the blog. Releasing the results to a private room audience days before you mention it publicly is the exact opposite of the heavy marketing you are speaking about.
The reason why it "feels" like heavy marketing is because SEOMoz has built up such a strong reputation and following that the slightest detail they release is immediately spread throughout the marketing world.
Would you prefer that SEOMoz stop working on projects like this and instead, we can go back to 2 years ago where everyone was just trial-and-fucking-error?
I disagree about not hyping preliminary results. I prefer to follow the research as it's being done, warts and all.
Protip: don't make business dicisions based on preliminary research results, no matter how "hyped" they are.
maybe they need a special tag for "preliminary results." I would much rather take place and hear the discussion good and bad as it happens. I learn more.