How A/B tests work

The 'subject' line of a newsletter and the 'from' name are two of the most important factors influencing whether or not an email will be opened. A/B split tests allow you to determine which of two different 'subject' and 'from' lines are more effective.

After creating a message with two different 'subject' or 'from' lines, FreshMail will divide your recipients into three groups: two smaller and one larger. The two smaller groups are sent the two versions that you set up. The messages have the same text and content, just different 'subject' or 'from' lines that appear in your subscriber's inbox. Our system then determines which message version is more 'successful' - that is, which one was opened more or which one gathered more clicks on your links. The 'winning' message version is then sent to the third, larger group of recipients. This ability to get feedback about the use of different words and phrases will help you to maximize the effectiveness of your campaign.

How to perform A/B tests

Tests can be performed after enabling the optimization tools in Parameters (the first step of the campaign creator).

After initiating an A/B Test, you will then have access to the settings. First you need to specify the size of the sample, i.e. how many subscribers are going to be included in the test group. Hold and drag the slider in order to set the required value.

When choosing the portion of your subscriber list that will receive each of the two test messages - A and B - the minimum for any campaign is 5% and the maximum is 25%.

The greater the number of subscribers on your list, the smaller sample that is needed. In other words, if you are sending your campaign to a very large number of people, a smaller percent of your subscriber list is sufficient to determine the 'winner' of your test. Bear in mind that in order to achieve more reliable results, it's best to get at least 50 responses from each of the two test groups before you have a clear picture of which message version works better for you.

What do you want to test?

In the next step, you decide which variable will be tested - the 'subject' line of the email or the 'from' line. After you make your choice you can enter the two variations that will be tested.

What determines success?

Now you choose between two options that measure the success of your campaign. It can be either the number of messages opened or the number of times a link in the message was clicked. Pick whichever is more relevant to your campaign.

How long should the test last?

The last parameter to determine is the duration of the tests. This is where you specify the number of hours after which the system will send the 'winning' version of the campaign to the larger third group of subscribers.

Now click 'Run the Test' and 'Next' to continue with the next stage of your campaign creation.

Done. Now you can move on to other things while FreshMail does your marketing research and proceeds with the results for you!