A/B testing text campaigns without fooling yourself
How to run meaningful SMS split tests — what to vary, how big a sample you need, which metric to judge on, and the mistakes that produce confident but wrong conclusions.
·5 min read
Split testing text messages is easy to start and easy to get wrong. With a list of a few hundred, the difference between two versions is usually random variation dressed up as insight.
Done properly it is genuinely useful, because SMS gives fast, high-signal feedback compared with most channels.
Test one thing, and make it a big thing
Change one variable per test, and pick one large enough to plausibly move the result. Swapping two near-identical phrasings wastes a send; testing a fundamentally different offer or framing tells you something.
The variables worth your time are the offer itself, the call to action, message length, and send timing. Fine-grained wording changes rarely produce detectable differences at the list sizes most businesses have.
- ✓Offer or incentive — usually the biggest lever
- ✓Call to action: reply versus link versus call
- ✓Length: short nudge versus fuller explanation
- ✓Send time: morning versus early evening
Sample size is where tests fail
A difference of a few responses between two groups of two hundred means nothing. If you cannot split into groups large enough to produce dozens of conversions each, you are not really testing — you are guessing with extra steps.
For small lists, the honest approach is to run the same test across several consecutive campaigns and look at the accumulated pattern rather than declaring a winner after one send.
Judge on the outcome, not the click
Reply rate and click rate are easy to measure and easy to mislead with. A message that gets more clicks and fewer bookings is not the winner.
Define the outcome before you send — booked appointments, completed purchases, closed deals — and measure against that within a fixed window. Also watch opt-out rate: a version that wins on conversion while burning subscribers is a loss over any real horizon.
Key takeaways
- →Change one variable, and make it a substantial one.
- →Small lists cannot detect small differences — accumulate across sends.
- →Judge on the business outcome, not clicks.
- →Track opt-out rate alongside conversion, or you optimise into churn.
Put this into practice with Text2Sale
Upload your leads, automate fast first-touch texts and follow-ups, stay 10DLC and TCPA compliant, and manage every conversation in one inbox.
Frequently asked questions
What should you A/B test in SMS?
Large variables: the offer itself, the call to action, message length, and send timing. Small wording changes rarely produce differences detectable at the list sizes most businesses have.
How big does an SMS list need to be for A/B testing?
Large enough that each group produces dozens of conversions, not a handful. Below that, differences are dominated by random variation. Small lists are better served by running the same test across several consecutive campaigns and looking at the accumulated pattern.
Which metric should decide an SMS test?
The business outcome — bookings, purchases or closed deals within a defined window — rather than clicks or replies. Watch opt-out rate alongside it, since a version that converts slightly better while burning subscribers loses over time.