I’m sometimes called upon to come up with subject lines and copy for campaigns, a task I enjoy and at which fancy myself pretty darn good.  But for one campaign we ran an A/B split test with the competing subject headings of, “Half-off for the Holidays” versus my “Everything half-off (even the partridge in the pear tree)”.  I was pretty confident I had the winner, but in comparing the results we noted that while the open rate was higher for mine, the more bland heading held a slight, but definitive advantage in clicks. Worse yet, the more bland heading had clearly resulted in more conversions to sales.

So what had happened?  There are many ways to interpret the data, but here are just a couple:

1.) People preferred the bland subject heading (I don’t believe it, but it is the simplest interpretation and we like to shave with Occam’s razor here at Sailthru.)

2.) People on that particular list are the type that like more simple subject lines (i.e. I didn’t know my audience.  It’s quite possible, but given the hip nature of the company sending the email I still don’t believe it.)

3.) Perhaps the clever subject heading had gotten some responders who wouldn’t have otherwise to open the email.  Ah!  Now we’re on to something.  (I’m not just saying this because it makes me sound better, I swear.)

Another way of phrasing this third interpretation is that those who clicked through and purchased were among the most engaged users of the site and were not heavily influenced by the cleverness of the subject line (or lack thereof).  If that’s so, then why had the simpler version resulted in more sales?  That’s not clear, though it is possible that given the list size results were skewed by one or two heavy buyers.

However, since our goal for that particular email was conversion to sales my rationalizations couldn’t hold sway and we went with the more simple heading.  But if our focus had been to build the size of our active list perhaps we would have run the other campaign; it had shown a small but decisive advantage in getting people to open the email.  Of course none of this takes into account the monumental importance that relevant and interesting content have on behavior once the email has been opened.  But that’s a topic for another posting.  Just remember, when running an A/B test take a close look at your analytics, they may reveal some interesting and unexpected behavior.  Even if it’s your own.

For detailed instructions on how to perform an A/B split visit:  http://docs.sailthru.com/ab_split

First a definition of terms: in it’s basic form an A/B split test is the competing of two versions of an email within a given campaign, each on a small percentage of a list.  Having monitored responses to each, the more effective of the two test emails is then sent to the remainder of the list.  It is important to remember that the two competing versions are run on exclusive segments of the list, that is, test recipients receive either version [A] or version [B], but not both.  The purpose and great power of an A/B split test lies in the ability to determine how your users are likely to respond to an email before having sent it to the vast majority of them.  Of course, coming up with two versions of a single campaign also puts to the test your basic assumptions about who your users are and how they will respond to a given message, thus making it a teaching tool as well.

Our system has a default setting of 10% for each of the A/B segments, which means that 80% of the list is withheld.  So, under the default settings, the winning email can be sent to 90% (10% test + 80% final) of the list (unless the final version of the email is a hybridized third version… So many options!).  You can specify any number of recipients for your tests, just remember that you want it to be a large enough proportion for the test to be meaningful, and a small enough proportion that the vast majority of the list receive the most effective version of your email.

A 50:50 A/B test is not really an A/B test
We sometimes get requests to run A/B tests on a different proportion of a given list.  The system lets you choose any fraction of your list that you specify.  But quite often we are requested to run a split of 50% and 50%.  As I said before, the system will let you do this, but just know that to do so defeats a central purpose of the A/B test.  After-all, once you’ve run your test on 100% of the list it’s too late to use any of the knowledge gained!  And even if you were to send the same email a second time you’d be in new conditions and sending to users who, at least half of which, had received the ad already.

For detailed instructions on how to perform an A/B split  visit:  http://docs.sailthru.com/ab_split