Remember back in January, when we asked you to help us run an experiment with the Marketing Experiments Blog testing the effectiveness of different email subject lines? The results are in, and we have a subject line winner! We'll talk about the test methodology and the winning submission, but before getting to that, I wanted to go over some of the common pitfalls and danger zones when it comes to email subject line testing (and, really, testing in general). Think of it like this:
(Image licensed from Getty Images)
Boundary #1: Make sure you're measuring the right thing
Generally speaking, the impact that email subject lines have on the performance of an email campaign is concentrated on open rate; more effective and intriguing subject lines drive more opens. This is because the subject line is the primary thing that you see when you make it to your inbox – and how much of that subject line a reader will or won't see is heavily influenced by that individual's choices in how they've set up their browser and reading panes.
Using my own email accounts as a visual example, you can see that the Gmail inbox can be generous; here it's showing up to 63 characters of the subject line and body text:
My Outlook web interface cuts at 52 characters, although this is heavily influenced by my setup – because my reading pane is set to "right," (vs. "bottom," or "off," Outlook's other two choices), I have less screen devoted to email previews and can see fewer subject line characters.
My Yahoo! Mail setup is the least generous, cutting subject lines at 49 charcters (but let's be real; it's unlikely that many of your potential customers are still using Yahoo! Mail).
If this is giving you the sneaking suspicion that email subject line length also has an influence on email subject line effectiveness, you're right. In our subject line test, we have line lengths ranging from 38 characters to 94 characters. The best performing subject line, in terms of driving the highest open rate? Smack in the middle at 51 characters.
Does this mean 51 characters is the ideal, maximum subject line length? Not necessarily. Too short can be an issue as well, as too few characters means fewer words at your disposal to entice an open and convey meaning. The three best performing subject lines in this test (average of 17.5% opened) averaged 51 characters long; the three with the lowest open rates (average of 15.9% opened) averaged 71 characters long. The two control group subject lines (average of 16.4% opened), at our shortest 38 characters, landed squarely in the middle in terms of open rate.
Boundary #2: If email subject lines only influence open rates, why should I track clicks?
An email subject line can also impact overall click-to-open rate for an email. This, by the way, is a better measure for performance than click-through rate alone: A high click-through rate but a lower click-to-open rate means that your body copy is strong but that you have opportunity to drive even more traffic by modifying your subject line for better open rates, thus increasing the size of the audience exposed to your awesome body copy.
A subject line sets up an expectation in the mind of the email reader of what is to come; how well the actual content of the email delivers against this expectation leads to either reader satisfaction or disappointment. Strong email subject line-content alignment generally leads to more clicks vs. a subject line that poorly represents the body content of the email.
I can illustrate this with an example of an email test that I ran years ago while working at an online travel company (without all of the specific numbers, which are proprietary), where we tested different subject lines offering varying percent discounts on the purchase of our products. Our test went something like this, but with a dozen or so different test cells sent to millions of customers:
- Subject Line 1: Get 15% off vacation packages!
- Body of Email 2: Blah, blah, blah, Get 15% off vacation packages!
- Subject Line 2: Open to discover your vacation package discount!
- Body of Email 2: Blah, blah, blah, Get 15% off vacation packages!
- Subject Line 3: [etc.]
What we learned was that we had better click-to-open rates on the emails where we had strong subject-body agreement, like in example 1; where we had vague subject lines we could drive a lot of interest (read: opens), but our body content seemed to disappoint in that our click-to-open rates were lower than in our matchy-matchy test cells.
For this VolunteerMatch email test, the body copy of all emails was identical except for one sentence; that one sentence had four different variations that were written to map to the six test (and one control) subject lines.
Our highest click-to-open rate (6.3%) in this email test, " Volunteering matters: We have the proof." was also the subject line that delivered the highest click-through rates (1.08%), even though it placed only second in terms of overall opens (17.3%). This indicates that the body copy of the email delivered on the promise of the subject line pretty well, and that an area of opportunity here would be to work on increasing overall opens (e.g., more potential people to click).
Our highest open rate subject line (18.2%), " The volunteer app your coworkers will talk about" did not win in terms of either overall clicks (0.98%) or click-to-open rate (5.4%). This tells me two things:
- The email body copy did not do a strong job of delivering on the expectations set by the subject line, and
- The more I can refine that body copy to closely match the expectations set by the subject line, the more likely I am to drive total clicks.
Boundary #3: Are you measuring or categorizing tangible things?
I call this the "specious lens" test. When you're looking at test results, be wary about what you use to classify or categorize your results. The subject line character length category is a tangible thing, perceivable by both testers and email recipients. Let's look at some other subject line classifications for this email test to see if anything else has a real impact on open rates:
- Use of special characters (e.g., punctuation marks)
- Use of title case vs. sentence case
Both use of special characters and use of case are tangible to customers. But from the chart above, you can see that there really isn't any correlation between either of these classifications and higher (or lower) open rates. The best performing subject line and one of the test's bottom three both excluded any kind of punctuation. Same for case; both the highest and worst performing subject lines used sentence case. Neither of these classifications appear to have any real, measurable impact, in this example, on customer email open rates.
If you are applying value categorizations to your test results, however, you need to be especially wary when trying to draw conclusions; this is because the value categories that you create are less likely to be tangibly perceptible by your customers. If I group the tested subject lines by the value or sentiment that they primarily convey, I create the following four buckets:
- Focuses on Caring as a sentiment
- Focuses on Mobile App
- Focuses on Quantifiable results
- Focuses on Values (Good/Bad)
If you are classifying your test results based on you or your team's value judgments, as I did here, and you can't see any performance difference between your classifications, as is true here, ask yourself, "Are these classifications tangible to the customer? Do they fail to have a real impact on outcomes, or are they simply not real?"
In this case, my answer is, "It's not that value or sentiment don't have an impact on outcomes, it's that these sentiment classifications are likely not perceptible to the customer and thus aren't a valid way in which to categorize this test." It's also risky to classify results after you already know the test outcomes; this can lead to you fitting a hypothesis to the test results vs. letting your test results prove or disprove your hypothesis.
Boundary #4: Statistics is your friend (i.e. math is important)
The last boundary to be aware of is statistics. Run all of your results data through some kind of statistical tool to make sure that the variations you're seeing between your test segments are more than just random background noise. There are a lot of factors that go into determining statistical significance, such as overall sample sizes, overall "action" rates, the differences between action rates, and how confident you'd like to be in your results (e.g., it's often easier to measure the difference between 1.1% and 0.1% than it is to measure the difference between 101% and 100%).
For this test, I've mentioned several times that two control emails were used. These both went to approximately the same number of people (36,000), and had identical subject lines and identical body copy. These two segments had similar, but not identical, overall open rates of 16.4% and 16.5%. In order to make sure that overall results are valid and there is no unintentional selection skew when creating (what should be random) segments, it's imperative to make sure that the variation between these two control segments is nothing other than random noise.
In the chart below, you can see that these slight variations in open rate between the two test cells are not statistically significant; a very important signal that the total data set from the test is valid, too.
If you don't have your own stats or analytical resources to help you with this last step, there are a lot of great tools and worksheets online to get you started, including the one that I've used here, from https://visualwebsiteoptimizer.com/
And now to the contest results!
The methodology
First things first, let's go over what was actually tested:
- 6 subject line "test cells" that each received a different email subject line
- 2 subject line "control cells" that received the same email subject line
- Just under 36,000 emails delivered to each test and control cell
- 287,117 emails delivered, overall
- Email body copy differed by one sentence in each test cell; otherwise was identical
Metrics recorded included:
- Emails delivered
- Email opens
- Email clicks
These three metrics were then used to calculate:
- Open rate (opens / delivered)
- Click-through rate (clicks / delivered)
- Click-to-open rate (clicks / opens)
The actual subject lines that were used in the test, along with all of the corresponding metrics:
Spread the Only "Good" Office Virus was used as the subject line for the two control cells (why use two control cells? The Marketing Experiments Blog wrote up their takeaways from the experiment a few weeks ago, and you can read the details and rationale there).
The winning, reader-submitted subject line (that drove the highest rate of clicks) was submitted by Moz Blog reader Jeff Purdon, an In-House Web Marketing Specialist for a manufacturing company. Jeff wins a ticket to the MarketingSherpa Email Summit 2015 and a stay at the ARIA Resort in Las Vegas. Congratulations, Jeff!
I think another layer of this that Email marketers do not take into account often enough is the amount of repetitiveness in their body content.
When you get me to click on your subject and then I realize that the body content is the exact same disappointing copy or link that I saw last week, then you have annoyed me even further and I may lose trust in you completely. Testing different subject lines with different groups is awesome, but testing different subject lines on the same person constantly will not produce desired results.
Joseph - agreed. Ideally, when you're testing email subject lines and/or body copy, you will be lucky enough to have a very large population against which to test so that you can keep each test send to unique groups of individuals. Repetitive testing against the same groups creates fatigue, and doesn't do favors for your brand.
Thanks for these tips, also for reminding us that we can break our leg skiing! ; )
Wow! I just started with email marketing with good open rates, but with this analysis and examples that you offer I think I'll improve the results, thanks!
According to me Email Subject live is most important aspect for email marketing. Mostly depend on it that someone going to open it to read, delete it or favourite it.
Hi Annette, as someone who has limited experience in terms of email marketing it is great to see that you can analyse email data to such a degree. Whilst larger volumes will give more accurate data there is always a feeling of subjectivity and nuance in regards to capturing people's attention - this article seems to point towards the opposite (with some compelling stats to back it up)!
"It's also risky to classify results after you already know the test outcomes; this can lead to you fitting a hypothesis to the test results vs. letting your test results prove or disprove your hypothesis."
Too true - it is sometimes the danger of statistics in that you can fool yourself into making the numbers do what you want them to do rather than using them more objectively.
Thanks Annette
Very interesting your article Annette
I think that over duplicate content anywhere, the reader is mortified because being signed and receive an email from the seller, this email does not provide some kind of "help" or "solution". In conclusion mail must not be the same for everyone, there must be a kind of segmentation.
Okay, I really tried not to chat back on this one, but I have no choice. I have a pet peeve, I'll admit it. I hate open rates. In my mind Opens are meaningless metric and most interpretations made off of open rates are also, yes, meaningless. I had the fortunate luck to have once led an email program that sent billions of emails a year - so a pretty good sample size - and because we built our own email platform, we could measure more things than most ESPs are willing to share.
First off, when you are looking at open rates, particularly across different email programs, what you are more likely to be measuring is the extent to which one customer set is using a email client that has image rendering turned on (pixel fire) vs. someone else whose client base made have a somewhat different mix. So its pretty darn hard to compare open rates across programs. Its also nearly impossible to measure your open rates over any length of time, particularly if your active account is churning at a decent rate.
Before anyone starts trusting the data they get back from their email server, I would highly suggest taking a look at how many of your email recipients that register a click do so WITHOUT registering an open. I've seen this rate vary significantly from program to program and within a program across time - and even with large list sizes and responses into then hundreds of thousands per campaign. For those with the computational power, I would even suggest looking at how the relationship between Click with Open and Click w/o Open varies across each of your users. Much of this can be driven by today's environment of users who read email across a plethora of devices and email clients, again some that are more likely to display images and others that do not. You might be surprised to find out that this variability often exceeds that 'lift' one thinks they are interpreting data from their aggregated test results. So emailer, beware.
Okay, so back to really cool email metrics. If you really want to look at very interesting way to measure email engagement, here is what I'd advise:
First gather the time stamp of when email was delivered, opened and clicked.
Segment out your users between those that have both and open AND a click and those that have just a click.
For the first group - put together a histogram that shows the differences between (open time less delivered time), (click time and open time), (click time and delivered time). For the second group you can only do (click time less delivered time). This is however a great way to normalize your real open rates vs. what pixel counting provides.
Next, if you are feeling really adventurous, slice the data based on the time zone of your end user (if you have that data) and see how response time changes with the local time of day. As you might guess, response time slows when you email either at the dead of night or when everyone else sends their Monday morning at 6 am email.
Really engaging email typically has short open-delivered time and longer click-open times. (e.g. they are actually reading your email and getting the hook)
If your subject line is really good - then you will see shorter times between open and delivery, longer time between click and open (although there are diminishing returns here), higher click to delivery and of course low bounce rates for paying off the message.
My 2 cents-
Marston
Hi Marston - great comments!
I, too, used to absolutely hate open rates (and frequently lectured against looking at them). I don't feel that way nearly as strongly these days - open rates have become much more reliable measures of success for (at a minimum) subject line effectiveness. This is because there has been a lot of progress/evolution within email clients, AND because new email clients (e.g., Apple Mail) have come into play that were not factors 5-7 years ago. Email clients today are much more sophisticated about how images are displayed, and much of the volatility that you mention (across email programs) has significantly stabilized in the last 1-2 years.
This particular test also didn't compare open rates across email programs -- it compared them across different segments of the same list, for email sent at the same time. So as a comparative measure, providing the segments were in no way skewed by domain or client (and two control segments were created to help detect any such skews), in this test open rate should be reliable as any skews introduced by lack of pixel firing should be spread evenly across all of the test segments.
I do love your comments on clicks w/out opens, time lags analysis, etc.; this is lovely if you can get to it. It's data that is unfortunately beyond the grasp of many, however, who do not have custom-built email systems at their disposal.
Ann - you might be surprised, but I know that Exact Target has the time stamps as well as data on clicks without opens. No doubt you have to arm wrestle the data out of the system. Most ESPs will accommodate custom reports for this info of you ask.
Great post Annette!
Optimizing an email subject line can have TREMENDOUS payoffs when done properly. It often goes overlooked as an inconsequential detail but it's really the first impression people get of your brand/company when checking their emails. For that reason, I'd say test, test, and test some more! You won't find a one-size-fits-all formula for subject lines but it does work to make it attention grabbing and pique the reader's interest.
I will add this info to my favorites ;)
Excellent article!
Both the email subject as the visible part of the body must create anticipation for it to open.Having more or less visible characters of the email body can be important if it is well written and captures the client to have interest in continue reading the email. Sometimes when you lose too short interest to click because the visible body of the email is not attractive.
This starts out really strong in that it clearly shows why subject length would matter by showing the limits on various platforms.There appears to be a typo. It looks like you meant Body of Email 1 when you wrote Body of Email 2 twice in a row in one of your examples.It finishes weakly in that it does not clearly state what the winning subject line actually was. That is information the reader has to kind of dig for by trying to parse the last chart.The section called Boundary 3 could be written more clearly to spell out the fact that there might be other things going on that we may or may not be able to readily and clearly categorize and measure. Plus, dealing with the issue of spam prevention needs addressed by the major public email platforms. All else aside, the analytics issues of measuring results was made very clear which is the most essential part of planning future email campaigns.Thank-you!
Thanks, this article has certainly got us thinking about the importance of subject lines!
We're curious about your throw away comment about no customers using Yahoo email though, we have plenty of Yahoo addresses on our databases. Why did you make that remark?
Yahoo! isn't in the top 5 email inbox providers (worldwide), at least according to some sources. It's always a good idea to know your own customer base, however; depending on your geography and the demographics of your audience, you will likely skew differently than global averages. A good source for general information, like this, can be found here: https://emailclientmarketshare.com/
Regarding the math, I'd always measure hard conversions, so you can focus on conversions and value. A header that drives traffic (FREE! You'll be rich and famous!) might attract traffic, but not necessary conversions.
Olaf - hard conversions are generally the ultimate metric, but there are many components of an email campaign that contribute to those hard conversions. You can often optimize an email campaign most effectively if you break out and test each of those things separately: subject lines affect open rate; body copy (and alignment between body copy and subject line) affect click-through rates; and strength of your product/offer/and landing page design affect conversion/sales rate.
In this particular test, the highest open rate email didn't have the highest rate of clicks; and the second highest ratio of clickers over openers was from an email whose open rate wasn't in the top 3. This means opportunity - clicks/conversions that might have been left on the table.
All that said - you need to have the time and the audience scale to break out tests like this!
Great tips Annette, what I have personally observed in most of the scenarios is the Email Subject line matters the most, the open rate heavily depends on it. I have done quite a bit of A/B testing with different subject lines and body but the only thing that showed me good difference was the subject and there was marginal to no difference with different body content.
Super post, Annette, and thanks so much for sharing the insight!
Thank you very much for explaining detailed thing about email marketing. i am regular user of mail chimp but i am not getting the result as i expect from email marketing. i think if i apply your ideas i might get some success. thank you again.
You mention about different subject line length in web browsers but surely mobile is important here. Aren't most emails now seen on mobile first?
Also you say testing against a control sample - which is great and what I always do / recommend but when you was looking at your stats did you take into account what device / browser etc they were reading the email in.
I agree about also tracking clicks rather than just opens, if your just tracking opens - click bait headlines will always win, but might not lead to clicks and sales. It's ok having a great open rate, but open rates don't pay the bills.
Another good thing to track is u subscribes - is a particular style of subject line leading to more disappointed users who then unsubscribe from your emails and shrink your potential customer base.
Overall great article on email testing. Just remember that famous phrase about stats, "there are lies, dam lies and stats". Stats can be changed to say what you want them to say.
Regarding mobile - if your customer base is mobile-heavy (and you can determine this by looking at your web analytics data to see how much of your site traffic comes from mobile devices), then yes, mobile will be another important consideration for your email tests. At Moz, 17% of the traffic to Rand's blog is from mobile; 13% of the traffic to the Moz blog is mobile; and 7% of our overall site traffic is from mobile. And Apple mail shows less than 40 characters of a subject line on my iPhone 5, in portrait orientation.
I'm also a fan of measuring the metrics that are important to your business. Unsubscribes is almost definitely one (although not looked at in this particular test).
Good tips here. I think the most dangerous thing for email marketers is indifference and apathy. You can send out an email to 100 people, but if only 10 open and just 2 click, you get frustrated. That leads a lot of people to quit, or at least think that email lists are unimportant. This is a dangerous route to travel down.
Best to keep at it - sometimes you just get useless people on your list because of your tactics to get list signups. Let's face it, many list signups can be utterly worthless, and will create the above problem.
Greg - I'm not a fan of actively creating incentives for people to sign up to an email list -- as you note, those sign ups end up leading to non-engaged email recipients. Also, if you keep your email content really timely and relevant to your customers' needs (read: don't just send an email because you haven't in a while -- make sure that you have something valuable to say) lack of engagement is much less of a problem.
Added to my bookmarks. Really nice post about email marketing. Keep up the good work!
The part I like the most about this post concerns the way that everything should be backed by numbers. They are, after all, a far more reliable metric than an email marketer's gut instinct.
Very helpful post for someone like me who's just getting into email marketing. I especially love the tip about making sure the body copy of the email delivers on the promise of the subject line.
Thanks, we are looking for ways to get our newsletter performance improved and this post plus comments to it will for sure help. And after reading it becomes also clear we need to pay more attention to monitoring after each newsletter is sent
Thank you for this nice article!
I use a lot for my emailing marketing campaigns and to make known my site. I realize now that it takes a bet on the subjet mail to have good returns.
Dear Annette,
Thanks for the nice article on email marketing.
Do you think sender name plays any important role in open rate?
I think USP of the product / services or important message placement in the subject line also needs to be tested?
USP of the product / services or important message / keywords at the beginning or visible part in the subject line irrespective of the length of the subject line (lets say first 40 characters) will definitely help in improving the peoples attention.
I think sender name is absolutely something that you can test, and the sending domain component of the sender name is a strong trust signal to both your customers and to the big ISPs.
Spaming emails its banned in Spain. Cops don't let us send it.
There was actually no mention of spamming in this post, unless I misunderstood something?