What Is an A/B Test in Direct Mail?

Key Takeaways

• An A/B test in direct mail tests multiple versions of one major element of a campaign is changed. Each version is then sent to a similar audience, and the results are then measured to analyze which campaign performed better.
• An A/B test in direct mail can be used to hone a specific part of a campaign, but multiple parts of your campaign shouldn’t be tested at once.

Introduction

When trying to improve a direct mail campaign, it can be hard to isolate one area of the campaign to analyze it. If something is working, why is it working? What about that part of your campaign makes it good? What about if something in your campaign is not working? What could you change about your campaign to improve it? To experiment with finding what works and what doesn’t, what is known as an A/B test can be conducted.

What Is an A/B Test in Direct Mail?

An A/B test in direct mail is when one part of your campaign, be it an offer, your creative, your postcard or envelope, or something else is experimented upon, and then sent to the different individuals in the same audience to see which mail piece performs better. This can be done with almost every part of your mailing campaign, but the key to an A/B test is to keep everything that isn’t being tested the same. This is to make sure that, when one campaign performs better than the next, it’s obvious why. If multiple aspects of the campaign is changed, then the campaign manager won’t be able to tell what worked and what didn’t.

How Do I Measure My A/B Test Results?

A/B test results are simple to calculate. Before sending out the different versions of a campaign, know which metrics are going to be tested. Depending on the goal of the campaign, a mailer might compare the response rate, conversion rate, cost per lead, cost per acquisition, customer lifetime value, return on investment and average response rate. After the mail has been sent out and the campaign has concluded that is when to check the results to see which version of the campaign performed better, and use those results to improve your next direct mail campaign.

Example

As an example, let’s say I’m sending out an offer to a community of 100,000 people regarding my new restaurant. The audience that I am mailing two is randomly split in half, so that I have two groups of 50,000 people, each group receiving one version of my mail piece. Both versions of my mail piece include the same offer of a 20% off coupon, are sent using the USPS Marketing Mail, and they were mailed on the same date.

The only difference between the two versions of my mail piece is the envelope that held my offer. One envelope contained just the restaurant’s name, whereas the other version had the restaurant name, our location, and a picture of the front of the building. I am planning on seeing how many people use the coupon to measure which campaign is a success. One month after the mail pieces started to arrive, I found that Version A of my mail piece, without the additional envelope decor, has 500 coupons redeemed, whereas Version B with the address and picture of the restaurant has 1,000 coupons redeemed, making Version B the winner.

Version A: 50,000 mailed → 500 coupons redeemed = 1% redemption rate

Version B: 50,000 mailed → 1,000 coupons redeemed = 2% redemption rate

Due to the envelope’s design being the independent variable that is being changed and tested, the results provide enough evidence that Version B’s envelope was more effective for this campaign.

What Can I A/B Test in Direct Mail?

You can test several different parts of a mail piece with doing an A/B test in direct mail, including:

• Headlines
• Offers
• Calls to action
• Images
• Copy
• Personalization
• Envelope designs
• Postcard designs
• Mail piece formats

When doing an A/B test in direct mail, just remember that only one major element should be changed, otherwise the results will be difficult to read, as it won’t be possible to pinpoint which part of the changed campaign was the main catalyst that sparked the change in metrics.

FAQ

• Q: Why can’t I change more than one aspect of my mail piece when doing an A/B test in direct mail?
  • A: If you change too many parts of your mail piece, then how will you know what each part of the different mail pieces performed well, and which parts failed to make an impact? If you change too much, it will become impossible to know which changes to keep.
• Q: Can I do an A/B test in direct mail with more than two versions of a mail piece?
  • A: Yes, you can. Testing more than two versions is possible, but you would need a big enough audience to have enough people to test the mail piece. Each group needs to be large enough to generate a meaningful result, which can depend on factors such as the expected response rate, audience size, and campaign goals.

Conclusion

A/B testing can help campaign managers make changes and decisions based on actual campaign results, rather than just assumptions alone. By only changing one part of your mail piece at a time, sending each version to a comparable audience, and measuring the appropriate performance metrics, a mailer can discover which version of the mail piece worked better and use the results from the campaign to improve future campaigns down the road.

If you would like to learn more about tracking your direct mail campaign’s results, learn more about that here.

New to Direct Mail?

Are you running your first direct mail campaign?

Read Geon’s beginner’s guide for to direct mail here.

Return to Analyzing Your Campaign Results, or browse Geon’s Learning Center.