Mailing List Testing

Key Takeaways

If testing multiple mailing lists, try to isolate the multiple mailing lists as the only thing being tested. Keep the offer, mail piece, geographic area, timing, and other major variables the same.
Testing multiple mailing lists can be used to make sure that a certain mailing list is the best for your current audience, offer, and more.

Introduction

If you’ve done a direct mail campaign, it’s natural to want to have the best mailing list to support you and your campaign going forward. If you have used a mailing list and want to test if a different mailing list could perform better, how do you do that? There are several steps to ensure that you’re actually testing the mailing list and not getting false results that could give a false positive towards one mailing list.

What Is Mailing List Testing?

Mailing list testing is the process of comparing two or more mailing lists, audiences, segments, or selects under similar campaign conditions. The goal of a mailing list test is to determine which audience is going to perform better for a particular mailing campaign. A mailing list test might be used to analyze the performance of a new mailing list that a mailer wants to test the credibility of, or if a mailing list that has been used in the past has started to bring diminishing returns for the campaign manager. There isn’t a bad reason to test a mailing list but know that it can take time and money to do.

The main reason why a mailer might want to conduct mailing list testing is because they want to find the best list that fits their target audience to generate the most success. While it might appear that one mailing list is going to outperform a different one, mailers don’t know before testing which list is going to perform better.

A list that looks good in theory might be outperformed by a different mailing list. There are several reasons why one list might outperform another list, such as one list’s audience having different interests, purchasing behaviors, demographics, or engagement offers with the offer on your mail piece. Data quality might also impact performance if one list contains outdated or inaccurate information. It’s the mailing list’s owner’s job to ensure that the mailing list has clean, accurate data, in a process known as data hygiene. Data hygiene ensures that the information contained within a mailing list is accurate. If a list owner fails to upkeep their list, the list might gradually start to underperform.

How Do I Test Two Mailing Lists?

Keep the Other Variables Consistent

When performing any test, it’s important to, at the best of the tester’s ability, keep the other variables (the offer, mail piece, target audience, etc.) the same so the only thing that is being tested is the actual mailing list. If for one test a campaign manager tests out an entirely different mail piece, offer, target audience, and time of the year than a different mailing list, it would be impossible to determine why one mailing list performed differently than the other, due to so many aspects of the campaign being different from one test to the next. Learn more about how to use test results to improve future campaigns here.

How Large Should My Test Be?

There isn’t a definitive answer to how large mailing list testing “should” be. A test needs to have an audience that is large enough to produce enough responses or conversions to make the list comparison meaningful. What the appropriate quantity of a test audience is will depend on factors such as the expected response rate, what the mailing list’s available universe is, campaign budget, and cost to produce each mail piece. Mailing list testing with a small audience can produce results that appear dramatic but are based on too few responses to draw a definitive, reliable conclusion.

What Results Should I Compare?

When you do mailing list testing, it’s important to remember that the list that produces the highest response rate isn’t necessarily the winner. One list might have a response rate of 5% and produces 10-dollar customers, but the other list might only produce a have a response rate of 3% but will generate 50-dollar customers.

10,000 test names at 5% x $10 = $5,000

10,000 test names at 3% x $50 = $15,000

There are several results that should be analyzed when conducting mailing list testing, such as the:

Common Mailing List Testing Mistakes

One of the biggest mailing list testing mistakes is just analyzing one metric and calling the results based on that one piece of information. As mentioned, just because one mailing list generates a higher response rate, that doesn’t mean that that mailing list is automatically better than the other list that was being tested.

Other common mailing list testing mistakes include:

Changing the offer between lists
Changing creative
Mailing at different times in the year (i.e. mailing in March and in December)
Comparing only the raw responses rather than conversion rates and other performance metrics
Using a sample size that’s too small to learn from
Declaring a universal, permanent winner after one test

Can I Test More Than Two Mailing Lists?

Sure, you can conduct mailing list testing. If you have the budget and enough of a sufficient test audience for your campaign, you’re able to do multiple tests. Just keep in mind that dividing a fixed test quantity among more lists will give you far fewer observations for each mailing list individually. Make sure that each of your test groups is large enough to produce a meaningful, accurate result.

What Do I Do with the Winning Mailing List?

A winning test after your mailing list testing could justify your use of that mailing list, but conducting only one test shouldn’t be treated as definitive proof that that list will always perform better than the other list. Consider testing the list(s) again or gradually increasing the total quantity that you mail. Continue monitoring the winning mailing list’s results such as the audience, offer, timing, and other conditions of your campaign change.

FAQ

Q: Should I stop using a mailing list if it loses a test?
- A: Not necessarily. A mailing list that underperforms for one offer or campaign might perform differently with another offer or campaign. Consider the sample size and reliability of your test results as well as whether you’re expecting to use that list again for a different audience, product, service, or offer in the future.
Q: Is the mailing list with the highest response rate always going to be the winner?
- A: No. Sometimes, a mailing list might produce a higher response rate but lower customer revenue, whereas a different mailing list could have a low response rate but generate more income per customer. Response rate shouldn’t be the only metric that is being measured if testing mailing lists.
Q: Can I test multiple segments of the same mailing list rather than multiple mailing lists at once?
- A: Yes, you can. As mentioned, try to keep the other variables, like the offer, mail piece, target audience, etc. the same as you test the different segments. Learn more about mailing list segments here.
Q: If I can test multiple segments at once, what about multiple selects?
- A: Yes. Do the same thing as the segments, keep the other variables the same as much as you can to try to isolate the selects to ensure that they are the only aspect being tested. Read up on selects here, if you have any additional questions regarding them.

Conclusion

The mailing list that you use is one of many factors that can impact the performance of any direct mail campaign. Conducting mailing list testing under similar campaign conditions for each mailing list could give you useful evidence about which audience is a better fit for a specific offer. By keeping as many other major variables as possible the same when evaluating response rates, conversions, revenue, acquisition cost, or other metrics, a campaign manager can make more informed decisions regarding which mailing lists to use in future direct mail campaigns.

If you’re unsure of what mailing list to use for your mailing campaign, or are thinking of switching mailing lists, contact Geon today, and we would be happy to help.

Geon reviews your intended audience and mail piece to identify additional criteria that can help refine your mailing list selection. This helps us select names that more closely align with your offer and campaign goals.

Learn why mailers choose Geon Media.

How Ordering Works

1. Tell us about your campaign. Share your intended audience, quantity, and a sample of your mail piece.
2. Review your options. We help you choose a mailing list and any suitable selects or segments.
3. Confirm your order. Review and approve the final pricing and rental terms.
4. Receive your list. We prepare your selection and deliver you with a spreadsheet containing your mailing list.

Learn more about ordering a mailing list.

New to Direct Mail?

Are you running your first direct mail campaign?

Read Geon’s beginner’s guide for to direct mail here.

Return to Analyzing Your Campaign Results, or browse Geon’s Learning Center.