A/B Tests

SmartSender.io A/B Testing involves sending up to three different versions of the same Email/WebPush/Telegram Campaign to a group of contacts and analyzing which one performs better. 

It allows you to compare the different variants of the Email/WebPush/Telegram message you want to be sent to your contacts. You only need to change one of the components at a time to compare the effectiveness between the variants, and the winner will be determined.

This guide emphasizes the importance of A/B testing email Campaigns to optimize their success rates.

The terminology you may encounter in this article:

• Campaign – sending batch newsletters and promo messages. Contains the audience and content.
• Test group – a group of contacts that help to determine the winner of the A/B Test on the basis of their behavior.
• Target group – a group of contacts that includes the Test group and the contacts who will get the “Winner”.
• Segment – a part of the Contact List that, based on specific characteristics, allows you to split your contacts or products into different groups.
• Contact List – is a place, where you can store and manage all data of your contacts.
• Audience – anyone your business and/or brand engages with, either directly or indirectly. The audience is accumulated in the Contact List.
• Component – the part of the variant that is the key element of the investigation (content, subject, from name, launch time).
• Winner, winning message – The Campaign that performs the best. It could either be chosen automatically through click rate, open rate, or unsubscribe rate, or you can pick it yourself based on what data is most important to you in the report.

A/B testing can help you:

• Minimize the risk of running new Campaigns by testing them before making permanent changes.
• Make data-driven decisions and improve the effectiveness of the Campaigns.
• Find out which Campaign is working and which one isn’t, based on the actual responses of the contacts.
• Create more effective marketing Campaigns based on the insights you gain, resulting in more engagement, leads, and sales.

In this article, you’ll learn how to create an A/B test.

Things to know before you start:

• You can send the A/B test to the entire Contact List or form the Audience and send the Test for this group.
• The sample may not be representative if the group of contacts is too small. If you want to reduce the margin of error, you have to increase the number of contacts in the sample.
• 5,000 contacts are recommended for a Test group.
• IMPORTANT The A/B Test Target group is dynamic and is taken at the time of the launch of a Variant or Winner. For example, Tests are not sent to contacts who are part of the Test group but have opted out of receiving communications or are on the Blacklist or dropped out of the Segment.
• You may send up to three variations of the Email/WebPush/Telegram messages to a small group of your Audience.
• By testing only one component at a time, you can accurately determine which specific element has the most significant impact on your open rates and click-through rates.
• The max length of the A/B Test Title is 500 symbols.
• Recalculation is initialized every time after moving from Step 1 to Step 2 or by pressing the recalculation button in the info table.

Step-by-step instructions:

1. Navigate to the A/B Tests Management page (Menu->Outbound->A/B Tests).
2. Press the “Create” button.

There are two steps in the A/B test creation.

1 STEP.

Step-by-step instructions:

1. Type in the title of the A/B Test.
2. Select the Contact List.
3. Choose a Segment. This is an optional choice, the A/B Test is sent only to those contacts that fall under the Segment rule.
   • You can create a one-time Segment by selecting the “Add filter” option. If you need more information on how to use the “Add filter” option, please refer to our article “Contact List Segmentation“.
     • NOTE The Segment will not be saved in the list of Segments. These groups of conditions only apply to that particular A/B Test.
   • You can change the filtering conditions by pressing the “Edit” button. Delete the created conditions by pressing the “Bin” icon.
4. Exclude Segments from sending. This is an optional selection, by using it you can exclude contacts from sending the A/B Test.
   • Here you can also create one-time filtering blocks. Learn more about the segmentation rules in our article “Contact List Segmentation“.
5. Select one of the available channels from the drop-down menu:
   • Email channel

     Email channel
     Components	Content Prepare your Templates in advance. Make one or two small changes, or design completely different templates. We recommend you test the efficiency of the CTAs, links, banners, images, buttons, text layout, etc.	Templates	Choose up to 3 Templates to test.
     	Email subject Test different Subject lines to find out which one is most interesting for your audience and which one prompted contacts to open the email you sent.	Template	Select the Templates that will undergo A/B Testing from the drop-down menu.
     		Email subjects	Enter up to 3 Subject lines for testing. Include global variables or emojis, and try different phrasing or sales offers.
     	From name Are you using the best From name for your Campaigns? Test it to find out!	Template	Select the Templates that will undergo A/B Testing from the drop-down menu.
     		From name	Enter up to 3 From names to test which personalization works best. Check if your recipients react better to emails coming from a person’s name or from the name of your company.
     	Launch time Learn when your customers are most active.	Template	Select the Templates that will undergo A/B Testing from the drop-down menu.
     		Send time	Set the date and time you are going to send the Template to your Test group with up to 3 different variants.
     Advanced settings	From name 'default'	Your clients will receive emails from this Sender. It can be the name of your company or a person’s name. NOTE Not applicable when you are testing the “From name” component.
     	From email 'default'	Your clients will receive emails from this email address.
     	Reply-To Name 'optional'	“To-Name” your customers will see when they reply to your Email messages. From name will be used by default.
     	Reply-To Email 'optional'	An address where you would like to receive replies from your customers. From email will be used by default.
     	Tags 'optional'	Tags help you in Analytics management. This field is likewise optional.

   • WebPush channel

     WebPush channel
     TTL (Time to live)	30 minutes	Defines how long the push service will attempt to deliver the notification. If the push service cannot reach the user within the TTL, the notification will be discarded.
     	1 hour
     	3 hours
     	6 hours
     	12 hours
     	1 day
     	2 days
     	3 days
     WebPush type	Both desktop & mobile	Select the type of Web Push notifications depending on which channels your Target group is subscribed to.
     	Only desktop
     	Only mobile
     Components	Content Prepare your Templates in advance. Make one or two small changes, or design completely different templates. We recommend you test the efficiency of the CTAs, links, banners, images, buttons, text layout, etc.	Templates	Choose up to 3 Templates to test.
     	Launch time Learn when your customers are most active.	Template	Select the Templates that will undergo A/B Testing from the drop-down menu.
     		Send time	Set the date and time with up to three variants to test when your recipients most likely open your messages.
     Advanced settings	Tags 'optional'	Tags help you in Analytics management. This field is likewise optional.

   • Telegram channel

     Telegram channel
     Telegram bot	Select the Telegram bot you want to use for the A/B test sending.
     Test component	Content Prepare your Templates in advance. Make one or two small changes, or design completely different templates. We recommend you test the efficiency of the CTAs, links, banners, images, buttons, text layout, etc.	Templates	Choose up to 3 Templates to test.
     	Launch time Learn when your customers are most active.	Template	Select the Templates that will undergo A/B Testing from the drop-down menu.
     		Send time	Set the date and time with up to three variants to test when your recipients most likely open your messages.
     Advanced settings	Tags 'optional'	Tags help you in Analytics management. This field is likewise optional.

6. The preview on the right side of the page shows you how all the testing variants will appear in the mailbox/device.
   • Use the arrow slider to check all the previews of the variants.
   • Email channel. The following information is also displayed for each of the variants:
     • From – the “From name” and “From email” data are displayed. Your client will receive the email from the provided address and name.
     • Reply to – the name and email address for which your client can apply for the reply.
     • Subject – the subject line of the email (the subject of the chosen Template. If the subject in the Template is not set – the subject becomes the same as the name of the Template).
   • WebPush channel. The preview on the right side of the page shows you how all the variants will be displayed in the browsers. Switch between tabs to check how your Template will be displayed on both: Windows and Mac OS for Chrome and Firefox browsers.
   • Telegram channel. The preview on the right side of the page shows you how the Template will be displayed for the recipient on his device.
7. Press the “Next step” button and you will be moved to the second step of settings.
   • After pressing the “Next step” button, the A/B Test gets the “Draft” status till you set the time and launch type.

2 STEP.

Step-by-step instructions:

1. Choose the percentage of your subscribers that will receive your test variants.
   • Set the desired percentage on the scale using the slider, and check the number of recipients that will receive your test variants in the Summary section.
   • You can also set the percentage of the Test group by pressing the “Edit” button.
   • The number of contacts you specify as a Test group is divided equally among the number of test variants. And the rest of the remaining contacts of the Target group are waiting for the winner to be sent.
   • The summary section displays the following parameters

     Contacts total	The number of contacts formed on the basis of the conditions from Step 1. It is the Target group.
     Test group	The number of contacts that will participate in testing. The data is displayed in two distinct formats: as a percentage and in numerical terms.
     Variant A	The number of contacts in each test variant group. The data is presented in two formats: as a percentage and numerically.
     Variant B
     Variant C
     Winning message	This data represents the number of contacts who will receive the winning variant after the completion of the test. It is presented in two formats: as a percentage and as a numerical value.

2. Define the criteria for choosing the winning message:
   • By open rate – based on unique opens.
   • By click rate – based on unique clicks.
   • By unsubscribe rate – based on the quantity of unsubscribed contacts after the test variants are sent.
3. Select the A/B Test duration. The winner will be determined based on the data for the specified time period:
   • 30 minutes
   • 1 hour
   • 3 hours
   • 6 hours
   • 12 hours
   • 18 hours
   • 24 hours
4. Define the actions with the winning variant:
   • Select “yes” and the system will send the winner automatically. Set the date and time to send the winner.
   • Select “no” and the winner will not be sent automatically. This option allows you to choose the winner based on reporting data or other factors you find to be the most valuable.
   • NOTE If the results of the tests are equal, the A variant will be sent by default.
   • If the “Launch time” component is selected, you need to select only the date for the winner sending because the time is already determined.
5. Select the A/B Test launch type.
   • Manual – you will need to launch the A/B Test by yourself from the A/B Test management page. Such A/B Test Campaign is created with the “Ready” status and waits for the start button to be pressed.
   • Now – the A/B Test Campaign is sent immediately after clicking on the “Launch A/B Test” button. It is created with the “Running” status because the sending process has already started.
   • Scheduled – a postponed A/B Test Campaign that will set off on the specified day and time. It is created in the “Waiting” status.
     • In case of testing the “Launch time” component, you will need to select only the date for the A/B Test sending because the time is already determined.
6. The table on the right side of the page shows you all the detailed information about your A/B Test Campaign which you can check before launching.

   General information
   Email	Content	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template A	The name of the Templates that will undergo A/B Testing. You can be redirected to the Email Template editing page by clicking on its name.
   		Template B
   		Template C
   		Subject A	The subject lines of the emails that will undergo the A/B Testing.
   		Subject B
   		Subject C
   		From name	Your client will receive the email from the provided name.
   		From email	Your client will receive the email from the provided email address.
   		Reply-to name	The name to whom your client can apply for the reply.
   		Reply-to email	The email address to which your client can send the reply.
   		Tags	The list of Tags of the A/B Test.
   	Email subject	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template	The name of the Template. You can be redirected to the Email Template editing page by clicking on its name.
   		Subject A	The subject lines of the email that will undergo A/B Testing.
   		Subject B
   		Subject C
   		From name	Your client will receive the email from the provided name.
   		From email	Your client will receive the email from the provided email address.
   		Reply-to name	The name to whom your client can apply for the reply.
   		Reply-to email	The email address to which your client can send the reply.
   		Tags	The list of Tags of the A/B Test.
   	From name	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template	The name of the Template. You can be redirected to the Email Template editing page by clicking on its name.
   		Email subject	The subject line of the email (the subject line of the chosen Template. If the subject in the Template is not set – the subject becomes the same as the name of the Template).
   		From name A	The From names that will undergo A/B Testing. Your clients will receive emails from the provided names.
   		From name B
   		From name C
   		From email	Your client will receive the email from the provided email address.
   		Reply-to name	The name to whom your client can apply for the reply.
   		Reply-to email	The email address to which your client can send the reply.
   		Tags	The list of Tags of the A/B Test.
   	Launch time	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template	The name of the Template. You can be redirected to the Email Template editing page by clicking on its name.
   		Email subject	The subject line of the email (the subject line of the chosen Template. If the subject line in the Template is not set – the subject becomes the same as the name of the Template).
   		From name	Your client will receive the email from the provided name.
   		From email	Your client will receive the email from the provided email address.
   		Reply-to name	The name to whom your client can apply for the reply.
   		Reply-to email	The email address to which your client can send the reply.
   		Tags	The list of Tags of the A/B Test.
   WebPush	Content	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template A	The name of the Templates that will undergo A/B Testing. You can be redirected to the WebPush Template editing page by clicking on its name.
   		Template B
   		Template C
   		TTL (Time to live)	Defines how long the push service will attempt to deliver the notification. If the push service cannot reach the user within the TTL, the notification will be discarded.
   		Tags	The list of Tags of the A/B Test.
   	Launch time	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template	The name of the Template. You can be redirected to the WebPush Template editing page by clicking on its name.
   		TTL (Time to live)	Defines how long the push service will attempt to deliver the notification. If the push service cannot reach the user within the TTL, the notification will be discarded.
   		Tags	The list of Tags of the A/B Test.
   Telegram	Content	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template A	The name of the Templates that undergo A/B Testing. You can be redirected to the Telegram Template editing page by clicking on its name.
   		Template B
   		Template C
   		Telegram bot	The name of the selected Telegram bot.
   		Tags	The list of Tags of the A/B Test.
   	Launch time	Channel	The channel type of the A/B Test.
   		Name	The A/B Test’s title.
   		Template	The name of the Template. You can be redirected to the Telegram Template editing page by clicking on its name.
   		Telegram bot	The name of the selected Telegram bot.
   		Tags	The list of Tags of the A/B Test.
   Winner variant settings
   Criterion		The specified criteria for defining a winning message.
   Duration		A/B Test duration. The winner will be determined based on the data for the specified time period.
   Winning message send time		The winner sending date and time. Required when the “send winner automatically” option is selected.
   Audience data
   Contact list		The name of the selected Contact List.
   Send to segment		Selected Segments for the A/B Test targeting.
   Filtered by		Information about the temporarily created Segment and its filter conditions.
   Exclude segments		Selected Segments that exclude contacts when sending the A/B test Campaign.
   Excluded by filter		The one-time filter, that excludes contacts from the Target group according to its conditions.
   Target contacts count		The number of contacts of the A/B Test Campaign. Press the “refresh” button in case you want to recalculate it one more time.
   Time frames
   Launch type		Displays the selected A/B test Campaign launch type.
   Launch time		The A/B Test starting time.
   Launch time A		The A/B Test starting time of different variants if the “Launch time” component is chosen.
   Launch time B
   Launch time C

7.  Manage A/B Tests. See how to do this in our article.