Jason Cohen recently authored a post on A/B testing that deserves both broader awareness and a deeper dive. Most of us in the online marketing world are aware of the power A/B tests can bring through improved click-through, sign-up and conversion rates. Getting a higher percentage of visitors to a page to take a desired action is powerful stuff.
The process by which we hypothesize, design, create and run testing, however, is fraught with peril. And, one of the least obvious, but most insiduous potential pitfalls is actually what we choose to test.
Visualizing the "Local Minimum" Issue
It's definitely interesting and sometimes worthwhile to test individual elements of a landing page, but it's often not appropriate at the beginning of a landing page or product's life. As Conversion Rate Experts points out, the "let's throw stuff at a wall and see what sticks" approach can have a small impact. Researching the questions visitors have and answering them effectively can make a world of difference.
The problem is, it's so very tempting to be seduced by an easy answer.
The Tantalizing Tease of Testing Minutiae
It's likely that many of you have read case studies like the ones below:
- Changing Button Color from Red to Green = 72% Improvement from Dan McGrady
- 11% Conversion Rate Increase with a "Commitment Checkbox" from Conversion Voodoo
- How Adding "It's Free" Increased Conversion by 28% from Soocial
- Human Photos Double Conversion Rate from Carsonified
- The infamous Twitter Following Command Test from Dustin Curtis
In all of these, some simple change accounted for big increases in click-through or conversion rate, leading to widespread praise and sharing. The problem is - they're the exception, not the rule. In fact, that's precisely why they're newsworthy and get so many mentions. That's not to say you shouldn't read them or shouldn't take away value from the examples (you definitely should). It's just that the mentality of the small change can create a misleading mindset for marketers.
Very few websites have the experience of changing a button color or altering a headline or fiddling with some copy and seeing huge improvements in conversion rate. If you have good reason to believe you're an outlier, go for it, just be cautious - it's not just the fact that small scale changes can have less positive of an impact. They also cost time and resources that you can't afford.
Some Simple, Compelling Math to Keep You Out of the Weeds
Let's say you're pretty good at conversion rate optimization - A/B and multivariate tests are relatively easy for you to perform and you've got solid instincts around them. And let's also say that you get reasonably decent traffic to your landing/test pages - in the several thousand range each day.
Even under these ideal conditions, massive problems emerge.
Knowing that each test takes a substantial amount of time to get high confidence of accuracy and that smaller tests (with less needle-moving potential) take MORE time is a pretty convincing reason to start out with the big ideas and big changes first. But, it's not the only logic behind this. Let's say you find a page/concept you're relatively happy with and start testing the little things - optimizing around the local minimum. You might run tests for 4-6 months, eek out a 5% improvement in your overall conversion rate and feel pretty good.
Until...
You run another big, new idea in a test and improve further. Now you know you've been wasting your time optimizing and perfecting a page whose overall concept isn't as good as the new, rough, unoptimized page you've just tested for the first time.
It's easy to see how you can get lost in this process and frustrated, too. That's why my recommendation (and the advice I get from lots of talented CRO folks) is to start with the big ideas and big projects, nail down the grand plans worth testing, let your audience pick a winner and then try to tweak, tune and improve.
What You Should Be Testing
What do I mean when I say "big ideas" and "overhauls?" Luckily, 37Signals provided a terrific example yesterday with their Basecamp Homepage Redesign:
They recorded a 14% improvement from new vs. old and can now decide whether they want to try another innovative concept or start optimizing the little things on this version. And while the numbers don't sound as compelling as a few of the bigger ones from the small tests, I'd argue they're going about things exactly in the right way. Perhaps a "little change" to the old version would have improved things quite substantially, but with this new version, they've got a higher base conversion rate and can benefit from every change that much more.
Another great example is the case study Conversion Rate Experts did for SEOmoz itself. That test gave us a 52% improvement in conversion rate from the PRO landing page. As an addendum, in April of this year, we tested an in-house created, shorter, less story-like landing page that we all hoped would beat out the old long-form version. After a few weeks of testing, it lost out. Later this summer, we'll be trying something completely different in an attempt to beat our current best.
The process to follow for conversion rate optimization and testing was well described in Stephen Pavlovich's post - The Definitive How-to Guide for CRO. His PDF guide, in particular, made this dead easy:
Step 1 of 4 in the process
Follow those steps, don't get lost in the minutiae, and you'll be on your way to exceptional results - no thousand monkeys with typewriters required.
p.s. I'd also suggest checking out this long but worthwhile post on stats for usability (and A/B) tests.
I get this all the time... Let's test the button color or the size of the text or the font type... To be honest I believe that people who take the approach like this think of the visitors as some little monkeys that react on different colors or sizes.
People who visit our website are human beings, therefore complex. Yes, the change of the color of a button might change on the way that a page communicates, but the truth is that how and what it communicates has the impact on the users, not the button colors.
Therefore, I totally agree with your approach to focus on big ideas. From the way people react to tests versions you also get to learn so much about them.
"how and what it communicates has the impact on the users, not the button colors."
the only way you would know that is by testing the colours i.e. testing the communication media.
The whole point of testing is to simplify people's complex reactions down to the level of monkeys or robots.
Thats why its a quantitative and not qualitative endeavour and thats why we measure to have significance
Any posts about CRO is very welcome... and this one is somehow a "best CRO advices repository" because of all the links you provide (I ask you to feel 'guilty' for my loss of productivity in the next few hours).
A post to be bookmarked surely.
About if it's better prioritize small scale vs. huge A/B tests, I believe it is to be considered as when a shop makes changes to its products expositions. With the premise that is always good to maintain a constant look & feeling in order to not "shock" your returning visitor, it would be good to refresh the image & visual architecture of a web site every amount of months (12/18) in order to better the CRO and prioritize the big changes.
I'm curious to read the comment by our CRO mozzers experts (Dr. Pete, this is especially for you).
Any posts about CRO is very welcome
Hear, hear! I echo Gianluca's sentiment. I can't get enough of good CRO posts.
CRO is my favorite part of internet marketing.
Good advice - I think the most pertinent phrase was "they've got a higher base conversion rate".
Changing the little things is always good if you have the time and resources available but these should always be done once you've nailed down the bigger areas.
Common sense, but so often overlooked.
As well as keeping out of the weeds, I guess there's a certain 'grass is always greener' mentality. Of course, it's natural to want to improve your conversion rates but when there are often just minimal improvements available, I wonder if it can counter-productive.
For instance, if you spend a week testing a completely new landing page design and get a 4% reduction in your conversion rates, then you try a new design again and get a 6% increase, presumably you're going to be tempted to go third time lucky and try yet another design. And another and another...
At some point, your regular visitors are going to get frustrated and confused with all the changes to your site. Every time they hit your front page they see something different. Eventually they don't bother to come back to your site. So maybe your conversion rate is improving but your actual visitor volume is dropping. Just a thought.
I agree with you. Anyway, the good designers are not they supposed to know what is more attractive to the consumer? ( like the green button versus the red one )
A better design can convert more and it seems logical to me but as you say, if done too often it is not better.
The Basecamp example is interesting:
"They recorded a 14% improvement from new vs. old"
The thing is - this improvement is in *click-throughs* from the homepage to the plans & pricing page, not *conversions* to the "thank you" page.
Basecamp should really run a follow-up test to see if this improvement is reflected in the conversion rate as well.
After all, there's no point increasing click-throughs if it doesn't also increase conversion. And it's even possible that a page that increases the CTR actually lowers the CR.
CRO is one of my favorite topics right now. That's one of the reasons we started doing web design and development - to get more control of the marketing and conversions. The best producer of ROI for an SEO campaign is increased leads and sales, and it is hard to get (what people consider) good SEO results without increasing ROI. SEO and conversion rates are getting increasingly connected and it's hard to offer one without the other.
Thanks for the visuals on where to expend energy. I love the basecamp example, and I have read similar stories in Website Magazine. This reminds me of the post (https://www.seomoz.org/blog/design-trends-the-single-purpose-homepage) about the single focus home pages. There is definately a trend in that direction, and basecamp is another example of knowing exactly what action you want people to take, and creating the entire page around that single focus.
I think there is a lot of branding that a company has to do before a single focus homepage will be effective, because if you don't have brand recognition, you have to use the valuable home page real estate to quickly and effectively educate your visitors as to who you are and what you do. That's slightly off the subject, but I thought it was worth mentioning :)
Thanks for the shout-out. Completely agree with your methodology, and agreed that everyone needs better knowledge about what "testing" means.
Interesting to see the Basecamp example. The new version just looks so much clearer and "uncluttered".
I always think less is more and try to ask myself "How much can I get away with NOT having on the page".
Cheers!
Jon
I think it depends. I tend to like less is more too. But, if people aren't converting because they want MORE information, then maybe less is less. Though the Basecamp change is more aesthetically appealing (to me) it also appears to better illustrate their value. I think your point is well deserved though, if clutter hides your value, then clean it up.
Exactly, always a fine line between over powering people with too much, and not giving them enough...
Logic is the driving force behind changes. It's good to see the concept of a broad change that can have a greater overall impact. Then, once the big change is made, little tweaks can be done. Thanks...
Great article!
We've recently A/B tested a totally new landing page at my startup and got a 50% increase in registrations.
Te new landing basically added a signup form on the right side of the page vs the old landing which had the classic "register now" button that took the user to the pricing page and THEN to the form.
We still have to check if the new registered users are as likely to become paying users as before, so it's still a work in progress.
Thanks!
Ya, I totally agree with the premise of this article. When I'm reporting on A/B testing case studies for WhichTestWon.com, I'm often drawn in by the hunt for those instances where a small change made a significant (10% or more) impact on conversions ONLY because our mission is to evangelize testing... so if we can show people a small change made a big imapact, mission accomplished. But, you're right in that most of the time it's the big overhaul that gets you the highest increase in conversions. I've seen it time and again.
Natalie Myers
Senior Reporter
WhichTestWon.com
Nice one - we have spent quite a while on this as well with our upcoming mobile app. Also, realized that mobile a/b testing is a pain, so we are going to launch new product around that!
This post couldn't have come at a better time. I'm actually right in the middle of creating our test page for our upcoming A/B test. tefal This helped a lot to produce ideas. Thanks!
Hm, I have to agree with "furstconversion" there - the cost/benefit calculation is what one should be looking at and immediately "statistical power" and "significance level" determination comes to mind. Not mentioned even once above.
Furthermore, the stats backing the graphs are dubious and uninformative, if not misleading. "Reaching statistical significance" is NOT a valid way to determine when to stop a test (statistically significant confidence makes no sense statistically, so I presume this is what you mean).
Having such a stopping rule exposes you to errors much greater than the ones your achieved significance level tells you. Furthermore, you speak nowhere of the effect size confidence intervals which is what would be the data one should look at when determining the success of a test. I bet those would look quite horribly if you stopped your tests the first time you reached significance (specifiying your level would have been nice).
For more on this -
https://blog.analytics-toolkit.com/2014/why-every-internet-marketer-should-be-a-statistician/
Hi Rand,another great useful post.In the upper area you wrote:Changing Button Color from Red to Green = 72% Improvement from Dan McGrady--> actually it must be "Changing Button from Green to Red" ;-)Thank you Petra
Great post. This is an area where I've made a lot of mistakes and learned these lessons the hard way. I recently shared some of my experiences in a 5 minute talk at the Boulder New Tech Meetup:
https://www.onlineaspect.com/2010/05/09/ab-testing/
Great post..!!
This post couldn't have come at a better time. I'm actually right in the middle of creating our test page for our upcoming A/B test. This helped a lot to produce ideas. Thanks!
Really liked your "Opportunity Cost" graphic Rand. It said it all in a nutshell.
My biggest takeaway form this is to really "count the cost" of doing A/B testing to ensure it will have a large enough payback for the dollars spent by the client.
It becomes even more critical when dealing with low volume websites that have to wait double, triple and even quadruple the time to be able to obtain statistically valid volumes.
For low volume websites it doesn't seem worthwhile to do this kind of testing. You could have a site up for two months collecting enough data only to find out you'd taken a 20% drop in conversions.
To be sure freshfish, you need to be extremely circumspect before embarking upon an A/B for a small site. You're right about having to wait a while for results.
Yet, at the same time, the designer can't know what the users would want in a site nearly as well as the users can speak for themselves.
That graph oversimplifies it.
For some sites the opportunity cost to create a 1% increase is minimal and that 1% is a massive gain (think what Amazon would do with a 1% increase)
The way we think about design today is wrong.
Accept that everything you design needs to be tested and you can save a lot of time and headaches by doing it upfront in the initial design stage.
If I was going to an Agency for a website today, I dont want to know what 1 site they build me, but what sites they build me.
Having a designer who is a front end developer saves you time and money because they design in a way thats easy to build. This will soon soon be the same for designers who design so its easy to test
By your logic 37 Signals' big redesign was also the exception to the rule. If ordniary people embark on a major redesign most likely they won't see large positive changes.
Simply saying success stories shouldn't be followed because they are the exception fails to consider the data and results from these stories actually contain.
Even though it's 'hard' .. math and visitor behavior always answers these questions for us.
Great post.
We are always told to TEST TEST TEST. And the reality is as you have stated above.
This is especially true for an organization that deals with many small clients (vs. just several large ones) such as ours.Our efforts, time and limited resources are better placed in other areas.
I am not saying not to test, but don’t split hairs over the minute details. Unless those minute details can actually correlate to BIG $$$.
Totally agree with you if you're using some manual testing tool (like GWO) where you have to keep manually changing code and updating your website by hand (that time could be put to much better use).
But with automated tools out there (such as the Conversion Chicken) that run and improve the site by themselves and don't require any maintenance there is little point in not testing, as they don't require any time investment and may lead to much better results.
Of course its great to do massive sweeping design changes (like 37 signals), but after that big design change is done why not test all the little things to squeeze out as many sales as possible?