The scholar Andrew Lang once said,
"He uses statistics as a drunken man uses lampposts - for support rather than for illumination. "
Unfortunately, I end up using statistics like a drunken man uses toilets - incorrectly and with my head hanging over the toilet-bowl thinking, "How did I get here?"
Community Analytics
With all the statistics available to online marketers, how do you decide what's useless and what's useful? Time to dredge up an old school metric called Signal to Noise ratio. SNR is a simple way to trend the health of the SEOmoz community.
Appropriated by the blogosphere from electrical engineers, SNR measures the quality of your user comments. It is relatively easy to compute - divide a user's MozPoints by their number of comments - and as will be revealed could quite potentially unlock the mysteries of the SEOmoz universe. I use 'potentially' because, like all well-intentioned drunken ideas, I have to admit I haven't really thought this all the way through.
Daniel and Rae's F%^king Excellent Adventure
My first task is to calculate the SNR for SEOmoz' top 100 members (I excluded the staff members because they're paid to play on SEOmoz):
After I'd done so, it doesn't take long to realize DanielTynski is a hidden gem. At 102 points and just 16 comments, he may be a man of few words but when he speaks, we listen. Daniel boasts an amazing 6.38 SNR, more than twice that of the next best. Daniel must be the epitome of the SEOmoz community, right?
Mayhap, because on closer inspection he has also submitted 3 (excellent) YOUmoz entries - one which generated 41 points - 40% of his total MOZpoints. Are YOUmoz points distorting the SNR?
Yes! Why? Um, I'm not sure, is there such a thing as analytics goggles? When I was drinking, SNR was looking a lot sexier.
Anyway, as my brain stumbles out of its stupor, I adjusted the original set of results with a little normalization to produce a handy weighted SNR.
With weighted SNR, Rae has everyone beat, coming in at 2.4 SNR (just ahead of Daniel at a very respectable 2.2). This immediately teaches me one thing.
Swearing a lot is the key to racking up MozPoints!
Welcome to drunken analytics, solving mysteries of the universe one at a time.
And If You Don't Know, Now You Know
So now you how the metric works. What else can the notorious S.N.R show?
Stuff.
SEOmoz' top 100 members (weighted SNR in brackets):
- SNR = 1.55 (1.44)
- Excluding staff = 1.35 (1.23)
- SEOmoz Staff only = 2.08 (2.08)
- Premium Members only = 1.49 (1.35)
Premium Members have a higher signal-to-noise ratio than regular members. That's a great reason to sign up to Premium, right Rand?
SEOmoz top 10 (from SNR100, excluding staff):
- DanielTynski
- sugarrae
- shor
- Igor Mordkovich
- stever
- Bud-Caddell (who inspired this post)
- Nathania Johnson
- feedthebot
- ciaran
- WebGeek
SEOmoz top 10 (weighted; excludes staff)
The comments of the top 10 are typically interesting and add value to the conversation (except for that shor dude, he talks about toilets).
SNR Head vs. Tail (excludes staff):- Head, top 20 members: SNR = 1.43 (1.27)
- Tail, top 500 members: SNR = 1.29 (1.18)
Weighted SNR of Top 500 members, including staff:
Note the concentration of high signal users in the top ranks and, more importantly, there are very few users below the lower control limit - the SEOmoz community is healthy!
SNR is a simple metric. Some would call it silly (just not to my face). It can't adjust for articles and comments predating the MOZpoints system, nor does it include SEOmoz staff entries. Should we refrain from engaging in banter, me-too comments and taking the 'bad cop' stance because of these results? No, because the community would become narrow, single-minded and suffer for it.
What do these statistics actually reveal? A lot. Or maybe nothing. As they say, torture numbers and they'll confess to anything. It is up to you to decide what the key performance indicators for your community are. Once you have these KPIs, my best advice is to set up those nifty upper and lower control limits to filter out the statistical noise from the signal.
The moral of the story? We might never invent a hangover cure, but at least Avinash Kaushik has analytics covered.
To paraphrase:
Signal minus Noise -> Insights -> Action -> Happy Community -> Traffic, Customer Satisfaction, Money!
How do you deal with signal versus noise?
What metrics would you suggest to measure the health and quality of your blog or forum community?
I like the concept of this post.
That said the MozPoint system -- like any point system that displays its plus/minus -- is more of a cult of personality than indicator of post value. It shows who is active and well liked, it encourages people to glad hand and make jokes (noise) in order to be well liked (earn points).
SugarRea has a great SNR, according to your system, but look at her recent comments. They are mainly just paling around; only one out of five is ostensibly related to marketing. I love some of her comments, but Rea’s comments frequently get multiple thumbs up on unrelated one-lines (many of the fun community members benefit this way). She is like The Rock, fun but I can’t really point out the last time she was in the ring.
Compare that to Michael Martinez, who has more thumbs down than the top 9 combined. His recent comments are 80% about the actual function of Google – and he gets a lot of thumbs down for it. Regardless of how you feel about his presentation he is largely on topic. He is like the Macho Man from early WWF – galvanizing, fun to watch, and always willing to get in the fight.
Knowing what the fuck you are talking about may help. A great one-liner is worth a dozen paragraphs in illumination.
Well said (does that count?).
*someone* was paying attention to the conclusion that Swearing a lot is the key to racking up MozPoints!
I'm not that sure I agree that the concluson was simply hot air for a link either ;-)
You're forgetting a category: some of us are neither funny nor clever. Never underestimate the power of stamina ;)
You are right Will, but I don't know which WWF member to compare you to.
Ok, first things first... I'm so anal about the spelling of my name... it's Rae (like a ray of sunshine, only I'm definitely not) and not Rea. ;-) However, I actually appreciate you doing that, because since it's the second time I've seen it accidentally spelled that way this week, I went and grabbed the .com for it.
I am indeed a one liner a lot of times, mainly because a lot of times, one line will do. And sometimes my comments are re: industry topics and sometimes they're just for fun. All work and no play would be - well, extremely unlike me. Additionally, I actually *read* all the previous comments and if someone has already posted sentiments close to my own, I don't see the need to regurgitate what they've already said in my own words to look smart. ;-) Since I typically get around to an seomoz post after it's already at 20 comments, that happens a lot.
However, I did take offense at one comment you mentioned:
>>>"She is like The Rock, fun but I can’t really point out the last time she was in the ring"
Maybe not in the comments at seomoz, but I'm in the marketing ring each and every day at all times. The above, undoubtedly not meaning to, implied that I'm some sort of laughs factory without substance. In addition to the fact that I do occasionally make posts of substance in comments here, I'm behind the scenes a lot too - contributing to surverys, post topics of the writers here as a result of ramblings on my own blog and even at times functionality of the seomoz site itself - that benefits the seomoz community as a whole.
I'd compare myself more to Randy Couture in regards to being a commentor on seomoz if we're going to go the fighter route. Never assume because you haven't heard much from me in a while (via seomoz comments anyway) that I won't kick some bloody ass when I do decide to open my mouth and hop back into a seomoz commenting ring.
...and sometimes a small book will do. ;-)
(well said Rae)
Gotta love UFC, well written Rae!
Did you link to Wikipedia just to piss the Mozzers off?
New material for Mr Wolf? "That's Rae, not Rea, R-A-E!"
"The above, undoubtedly not meaning to, implied that I'm some sort of laughs factory without substance"That's why drunken analytics is such a big industry. It is very easy to get mislead by what appears to be 'hard data'. If you looked at Rae's recent comments you might make the same mistake as inflatemouse when in fact, for every one-liner Rae makes, there is a value-adding comment with 4,5,6 mozpoints etc.
Besides, my joke about swearing was just a cheap gag so I could link out to markus' game ;)
Sorry about the misspelling your name. And on the Rock comment I was speaking purely in the context of your Moz comments (because Shor indicated you as the highest weighted SNR). I’m not the type of person who would talk about you on a professional level, as I have never personally worked with you.
I want to reiterate that I enjoy your contributions and think you are a valuable member. If you look back at my year and a half of comments you are as likely to find commentary on the facial hair or sandwich stealing as you are to find a comment on web development. This place is fun because it is NOT all business.
The only thing I meant to imply is that MozPoints are independent of a commenter’s tendency toward Technical vs. Community comments.
Interesting post. I think the quantity of comments has greatly increased since moz points started. (Looks like another statistics article for you to do.) I remember seeing my rank shortly after moz points was launched and I was in the top 50--don't remember exactly where. I quickly moved to nearly 100 since I don't comment as much anymore. If you are looking for moz points and a higher rank there is an incentive to make "nice post" comments. Almost all my comments are pre-moz point launch which means they are all worth one point. Who is going to go back in the archive and mod up comments? I think that most of your high scorers have probably made most of their comments recently. As for me...without my Digg comment article, my SNR would be pretty close to 1.
I agree with Tom6a.
If you don't count points in posts, then the mozpoints are pretty much equal to the amount of comments for most (I guess).
I see thumbs down as an indicator of quality or at least a contradictory comment.
Personally I think my comment who got 2 thumbs down was the one that added most value.
I must say I was surprised to see I ranked in a top position. I honestly regret not posting more comments than I do....
More than anything, I like to share information/articles here because I know the SEOmoz community will really appreciate it. This great report gives me some motivation to start contributing more on the comments side of things!
It is good dialog, afterall, which encourages the types of questions that result in new and interesting content being generated.
I am a drunk and as such, your article is acceptible since you mention the word "drunk" and I am mentioned.
If a SEOmoz article mentions the word "drunk" and I am not included, expect problems.
This article very probabaly has value, it is written by shor, but I am afraid that I am drunk and unable to appreciate things like "value" at the moment, but I saw my name and I like that.
Ok, so I totally get that the article was mostly about some fun, but there are also some useful ideas in here worth exploring a little more
How did you choose the "weight" in the weighted SNR calculation? You mentioned something about normalising, but I can think of a handful of different ways of approaching this, and they could easily give quite different results. I fairly regularly have clients (not SEO clients!) who want to use "an average" or to be "more scientific" a "weighted average". Weights are important, and usually make most sense when there is some real hard stats behind the choice. As soon as it becomes a subjective weighting of one thing as "more important" than another, the results can be tweaked to get almost any result you want. Dangerous when certain result are more desirable than others.
The fact that premium members have a higher SNR than regular members more likely means that:
It would also be interesting to put a multivariate model on the SNR. You have already established that different "class" of member (staff, premium, regular) have different SNRs on average.
As always, give me the data and I'll gladly play with it and post the results back on SEOmoz.org
“How did you choose the "weight" in the weighted SNR calculation?”
I am amazed I could do this while tipsy but I believe I found the average number of YOUmoz comments per YOUmoz post and after rounding, it came in at ~10. Next I adjusted the MozPoints results by weighting down the value of YOUmoz entries. Not too scientific, but my objective at the time was to show the SNR of comments, not the SNR of [comments + YOUmoz entries].
“The fact that premium members have a higher SNR than regular members more likely means that:”
Agree with first two points. 3rd bullet: there is definitely a branding effect. A throwaway (eg. me-too) comment from a respected member or staff often receives multiple thumbs. In other words, build your personal brand! 4th bullet: I didn’t look at statistical significance at all, a bit of an oversight I guess.
It would also be interesting to put a multivariate model on the SNR. You have already established that different "class" of member (staff, premium, regular) have different SNRs on average.
Timeliness data would require access to the SEOmoz backend or else a script would have to be written to crawl the comments! Other interesting data to play around with is the connectivity between members – eg. who is voting for who and looking for ‘cliques’. Social voting sites like digg are really leading the field in this type of community analytics.
Thanks for the deeper analysis David!
Shor,
It would be interesting to see these modifications in place if you ever decide to do a "Drunken Statistics Part Deux". :)
wow! i had no idea...never thought about mixing drinking and analytics together before. i'm gonna have to try that out.
very interesting post. and thanks, shor, for the ideas. (and for placing me in top 10!)
Analytics often drive me to drink, but that's a little different.
Glad other people have these crazy ideas too and it's not just me! I thought about writing something just like this.
I'm honoured to make the top 10 of one of your lists ;)
Whoo-hoo! #10 on the Weighted Top 10. Take that, Sally Johnson from 2nd grade! Who has cooties now?!
As someone who's taken too much statistics in his life, I'm both intrigued and have no idea what this means exactly. I mean, some conclusions are obvious: clearly, sugarrae is da bomb, for example. I know I personally got a lot of mileage from one of my YouMOZ posts.
I have seen a certain popularity effect. If Matt Cutts posted verses from Lewis Carroll's Jabberwocky, he'd probably get three thumbs up. Of course, he doesn't grace us with his presence that often, so he didn't make the list.
Heh! and being the geek that I am I had to go and work out my SNR didn't I ;-) 2 things I have to copy here because they are both absolute gems:
"He uses statistics as a drunken man uses lampposts - for support rather than for illumination. " - followed by your toilet analagy that I TOTALLY get, and while I love stats, they are badly abused, which is why many distrust them, making our analytics and reporting job that *smidge* harder than it should be, especially when trying to convince a client of the need for something based on statistical analysis...
and...
"The wrong people are buying our product in large quantities"
I realised some time ago not to give a sparrows fart which people were buying my product, especially when they were buying in bulk, just as long as they were buying (possibly a somewhat short-sighted approach admittedly), but so nice to see that the issues of targeting objectives veering somewhat off-course is more common than one might think! Great post in my humble opinion :-)
Wow...that's a great way to look at things. SNR - simple and insightful.
Great Post!
This was easily the most confusing thing I have read today.
What does the last graph's X stand for? There's an odd saw pattern there.
Shor, great post! Wow, you really dug into the stats! I have to say, I was surprised to show up in the top 10. :) It's funny, because when I started here, I remember reading some really good comments by others and feeling challenged to make worthwhile comments. I think it's a credit to the entire community here that the SNR of SEOmoz as a whole is higher than most forums or SEO communities.
Overall, a great read!
Being able to spot trends before they occur is best when you get that 'gut' feeling and then have some real stats to back up the feeling, IMO. Love the opening analogy shor, thanks for the insight.
....and shor climbs the SNR list with an excellent YOUmoz post promoted to the main blog...
The SNR makes for an interesting statistic. I think I might look at implementing a 'Karma' system of some sorts on some of my community sites so I can start tracking statistics of my member base - it is a decent way to distill who are your 'best' members in terms of contributions.
Thanks for the excellent write-up!
'....and shor climbs the SNR list with an excellent YOUmoz post promoted to the main blog...'
Yep, contributing further to Shor's favourite thing about SEO, though the inexorable climb would appear to have gained significant speed
:-)
Karma systems are a bit like WOW and Everquest, where time invested often trumps the skill of a player. That is, many karma systems tend to skew towards rewarding quantity over time rather than quality over time.
This in itself is not really an issue, as a lively and fast-moving community is preferable to a static one, but you should also pay special attention to both your 'community gurus' and newcomers. SEOmoz is doing pretty good at both.
Very interesting post. Neat to see myself on the top 10 list also ;)
Nice post.
Now the question is, did you plan on writing this post and get drunk in order to do it, or did yiou just happen to get drunk and have nothing better to do than statistical research? Any way you slice it, you once again managed to post something insightful, fun, usefull and more well-researched than most of us would take the time to do. Good on ya, Shor. Also,
Nicely done. "Super Nin-ten-do, Sega Gene-sis"
I don't remember.
Alternating between whites and reds the night was a haze but I eventually I built up the courage to walk over, look her in the eye and hit that YOUmoz compose button. ;)
How can you expect to survive in SEO without first getting dirty by jumping into stats!
In reply to your question over managing our own site SNR, I find by grabbing my highest page-view and visitor periods and then referring to articles written within that period I can determine what trends are popular and focus further on those specific topics.
For a period I was finding my humour posts getting far more positive response (volume of traffic and comments) than any SEO/SEM topic! Indicates how little demand by the greater market for our knowledge and consequent services.
Great article shor, very appealing language which I loved and very personal. That game rocks btw!
It's simultaneously funny and serious the day you discover your actual readers are not your targeted demographic.
Do you:
On one blog we began with three different types of blog entries; knowledge-based, tutorials and editorials. The editorials generated hundreds of more comments and page views - it was clear what our site users wanted, to talk and debate with us!
All of the blog's posts are now editorials and very opinionated.
I think Guy Kawasaki summed it up best during his Art of the Start presentation:
"Oh my god, the wrong people are buying our product in large quantities" ;-)
Thanks visser, I knew the Aussies would enjoy my drunken antics- err, analytics.
I am now completely, and utterly, confused.
You forgot to mention the coveted bottom spot, which belongs to the X.