This article will introduce you to the concept of Split Testing (also known as A/B Testing).
Before starting, you should be familiar with:
What is a Split Test?
A Split Test, or an A/B test, measures the performance of two variables in your email to see which works best for your audience.
For example, you might test two different subject lines to see which email gets the most clicks.
Each version of your split test campaign is sent to a percentage of your Customer List. For example, if you set it to 50% then 25% will get version A and 25% will get version B of the Email Campaign.
Once sent, the success of each version will be analysed and the remaining balance of your contacts will receive the more successful version.
You can split test:
- Subject lines
- Friendly from name
- Content blocks (creative)
When setting up a Split Test, you can choose which metric will determine success:
- Opens: the number of individuals who opened your email.
TIP: Apple's iOS15 or later, and, MacOS12 Monterey or later software updates mean that open metrics are an unreliable measure of success. We would always recommend using Clicks instead.
- Clicks: the number of individual subscribers who have clicked on the links in your email.
How to set up a Split Test
There are three steps to setting up a Split Test:
- Setting up variables in Dotdigital
- Creating variables: Subject line, friendly name or from address or Creative
- Sending the Email Campaign in Spektrix
How to set up variables in Dotdigital
Split Test settings are configured during the Campaign Details section of Setting up a Dotdigital Email Campaign.
- From the Campaign Details page, select SPLIT TEST SETTINGS:
- In the Split test settings side panel, toggle the Split Test features you want to use to ENABLE:
-
Subject lines: This is the introductory text of an email that is displayed in your customer's inbox.
-
Friendly name: Your friendly “from” name is the name that appears beside your subject line in the recipient’s inbox.
- From address: This is the email address which your Email Campaign appears to be delivered from.
TIP: The From address option only appears if you have more than one from address configured on your account.
- Creative: The content and design of your Email Campaign once opened.
Select APPLY to save your settings and create a split test email.
Subject line, friendly name or from address
To set up variations for subject line, friendly name or from address, navigate to the Create tab.
TIP: You can only add versions on the Create tab, not on the Content tab.
Above Campaign details you should see Version A and Version B:
- Set up the Campaign details for Version A and Version B.
- Select the + (plus) symbol to add more versions, up to a maximum of ten. Select the X symbol to delete a version.
- Then click SAVE & CONTINUE.
Creative
To set up two different versions of your campaign creative (the content and design of your Campaign), navigate to the Content tab:
Above Campaign details you should see Version A and Version B.
- Click into each Version to design the content.
- When both Versions are complete, then click SAVE.
TIP: When you switch on the Split Test Creative setting, Version A of your Email Campaign will be copied to create Version B. To save time, you can create Version A before switching on the Split Test setting. This will copy your content into Version B which you can then amend.
Sending the Email Campaign in Spektrix
As with other Integrated Email Campaigns, once you have designed your Campaign in Dotdigital you’ll need to return to Spektrix to send it.
Follow the steps in How to send an Integrated Email Campaign to set up your New Integrated Mailing:
Additionally, for Split Tests, you’ll also need to set a Test percentage, a Test period and a Test metric:
Test Percentage
Test percentage is the percentage of contacts to be included in the test.
For example, if the Test Percentage is set to 50% then 25% will get version A and 25% will get version B of the campaign. The remaining 50% of your selected audience are sent the winning version once the testing period is finished
- The default setting is 5%.
- Split test percentage can't be set to anything higher than 60% in Dotdigital.
WARNING: If you set the Split Test percentage to higher than 60% in Spektrix, Split testing won't work and Version A of the mailing will always send.
Test Period
The test period is how long the test will run, in hours.
During the test period, both versions of the Split Test will be sent to the test percentage.
You must enter a number higher than 0.
Once the test period has ended, the most successful version is determined based on the test metric. The rest of the contacts will receive the winning version of the email.
Test Metric
The test metric is how the winner of the Split Test is determined.
For example, if you choose to set the metric as Clicks, the version of the split test with the most clicks during the test period will win.
There are two Test Metrics to choose from:
- Opens: the number of individuals who opened your email. Best for analysing Subject Lines and/or Friendly Name and/or From Address.
TIP: Apple's iOS15 or later, and, MacOS12 Monterey or later software updates mean that open metrics are an unreliable measure of success. We would always recommend using Clicks instead.
- Clicks: the number of individual subscribers who have clicked on the links in your email. Clicks are the best measure for analysing Split Tests based on Subject Line, Friendly Name or From Address variations.
In a situation where a winner cannot be determined, version A is sent to the remainder of your contacts. This can happen if there is no engagement with your campaign during the initial test period. For example, if you have selected clicks as your metric, but no contacts clicked during the test period.
Evaluating your Split Test
After your Split Test has been sent, you can view the Campaign Report to see statistics about the send, and how each Split Test version performed.
To view a campaign report:
- Go to Campaigns > Email > Sent tab, or Analytics > Reports.
- Select the campaign name.
If your Split Test is still in progress, you can see how long remains before your Split Testing period has finished:
Further Reading
Next, you might want to learn more about Setting Up a Dotdigital Email Campaign or Reporting on Email Campaigns. You can find more information in the Communicate with your Customers section of the Support Centre.