Manage A/B Tests

Email marketing is an essential component of any successful digital marketing Campaign. However, not all messages are created equal. To optimize your Campaigns, A/B Testing is a must.

In this article, we will show you all the controls on the A/B Tests page.

To start work with A/B Tests – follow the steps below:

1. Navigate to the A/B Tests Management page (Menu->Outbound->A/B tests).
2. Press the filter button to select the channel.
   • All
   • Email
   • WebPush
   • Telegram
3. Press the filter button to select the status.
   • All
   • Draft
   • Waiting
   • Ready
   • Running
   • Done
   • Stopped
   • Failed
4. You can see the interaction of buttons and statuses with each other here.

   Button	Status	Launch type
   	(Ready for sending, can be launched, and can be edited before launch.)	Manual
   (The stop/disable button)	(Cannot be stopped and edited.)	Now Manual Schedule
   	(If you pause an A/B Test and try to restart it after the launch time has passed, an alert about the missed time will be displayed. You can still edit such a campaign.)	Schedule
   	(If the A/B Test was stopped before sending. Can be edited and launched.)	Schedule
   	(Cannot be edited and cannot be launched.)	Now Manual Schedule
   The A/B Test is not active	(If you leave the A/B Test setting page after pressing the “Next step” button. Can be edited. Cannot be launched.)	Not selected
   	(The reason can be seen in the corresponding table by pressing the “Info” button. Can be edited.)	Now Manual Schedule

5. Sort A/B Tests by name.
6. Sort A/B Tests by test component.
7. Search A/B Test by name and by ID.
8. Press the “Analytics” button. The report shows the performance of your A/B Test campaign and its key metrics.
9. The A/B Test Analytic consists of such parameters.

   Section	Parameter	Displays for	Description
   The A/B Test detailed information	Channel & Status	All channels and test components.	Shows the A/B Test Campaign channel and its status at the moment of the Analytics checkup.
   	Name	All channels and test components.	The name of the A/B Test Campaign.
   	Test components	All channels and test components.	The tested key element of the Campaign (Content, Subject, From name, Launch time).
   	Template	All channels and test components except for the “Content” component.	The Template that undergoes A/B Testing.
   	Template A	All channels but “Content” test component only.	The name of the Templates that are undergoing A/B Testing.
   	Template B
   	Template C
   	Email subject	Email Channel. For the “From name”, and “Launch time” components only.	The subject line of the A/B Test email Campaign.
   	Email subject A	Email Channel. For the “Content” and “Email subject” components only.	The subject lines of the emails that are undergoing the A/B Testing.
   	Email subject B
   	Email subject C
   	From name	Email channel. For all test components except the “From Name” component.	The provided name that is displayed in your recipient’s inbox.
   	From name A	Email channel. For the “From name” component only.	The From names that are undergoing A/B Testing.
   	From name B
   	From name C
   	From email	All components of the Email channel.	The email address from which the A/B test campaign is sent.
   	Reply-to email	All components of the Email channel.	The email address to which your client can send the reply.
   	Reply-to name	All components of the Email channel.	The name to which your client can apply for the reply.
   	Tags	All channels and test components.	The list of Tags of the A/B Test.
   	TTL (Time to live)	All test components of the WebPush channel.	The defined duration of the push service attempts to deliver the notification.
   	Telegram bot	All test components of the Telegram channel.	The name of the selected Telegram bot.
   	Contact list	All channels and test components.	The name of the selected Contact List.
   	Segments	All channels and test components.	Information about the selected Segments that determined the A/B Test Target group.
   	Exclude segments	All channels and test components.	Information about the filter, that excluded contacts from the Target group.
   	Filtered by	All channels and test components.	Information about the temporarily created segment.
   	Excluded by filter	All channels and test components.	Information about the one-time filter, that excluded contacts from the Target group.
   	Launch time	All channels. For all test components except the “Launch time” component.	The A/B Test starting time.
   	Launch time A	All channels. For the “Launch time” component only.	A/B Test start time for each variant.
   	Launch time B
   	Launch time C
   	Criterion	All channels and test components.	Defined criteria for choosing the winning message.
   	Duration	All channels and test components.	A/B Test campaign duration. The winner is determined based on the data for the specified time frame.
   	Winning message send time	All channels and test components.	The winner sending date and time. Displayed only for the “send the winning message automatically” option.
   The export report button	Excel	All channels and test components.	Downloads all data from the Results table in tabular form (no graphs) in XLSX format on your device. NOTE The most comfortable way for the downloaded file to read is to open it via Google Sheets.
   	PDF	All channels and test components.	Downloads data and graphs the way they appear on the page in PDF format.
   Winner determination	Winner variant	All channels and test components.	Displays information if the winner is not determined yet: NO WINNER YET- the test is still running; NO WINNER YET – The process of sending to the winner has been halted. Displays the name of the winner – the variant that outperformed all other variations. When the “Content” component was tested, then the displayed template name of the winner will be clickable.
   	Criteria for choosing the winning message	All channels and test components.	Previously defined criteria for choosing the winning message: By open rate – based on unique opens. By click rate – based on unique clicks. By unsubscribe rate – based on the quantity of unsubscribed contacts after the test variants are sent.
   	Test duration	All channels and test components.	The specified time frame, during which the winner is determined: 30 minutes 1 hour 3 hours 6 hours 12 hours 18 hours 24 hours
   	Winning message send time	All channels and test components.	Appears after the winner is already determined. Displays the date and time when the winning variant will be sent to the rest of the Target group.
   	Test components	All channels and test components.	The list of Test components that are objects of the A/B Testing: Content (the names of the Templates are clickable) Email subject From name Launch time
   Chart overview	Test components and Final dispatch	All channels and test components. NOTE Appears only if the duration of the test is three or more hours. NOTE Reflects hourly-based metrics dynamic, accordingly to the criterion selected earlier to determine the winner. Each bullet displays a data summary for the last hour.	Horizontal lines show you the dynamics of Test variants and the winner. By hovering the cursor on the bullets the following information is displayed: The Test time frame which has already passed. Test components. The number of contacts that have met the conditions of the winner determination as a result during the previous hour time frame. The Test group percentage for the particular Test variant. You also can view separately only one of the indicators – “Variant A”, “Variant B”, “Variant C”, or “Final dispatch”. Just click on the corresponding field to exclude the parameters view.
   Statistic type	During the test period	Al channels and test components.	This option is selected by default. The table below will show you only the data related to the test period.
   	During the entire duration	All channels and test components.	Select this option from the drop-down menu if you want to display all the data from the very beginning till the moment you check the Analytics.
   Results table Reflects all the parameters that occurred during the Test period or during the entire duration.	Variant	All channels and test components.	The list of the tested variants and the winning message. The winner is shown with an icon after being determined.
   	Recipients	All channels and test components.	The number of contacts who received test variants and the remaining contacts who received the winning message.
   	Delivered	All channels and test components.	The status is assigned as soon as SmartSender.io has sent a message and it has been accepted by the server of the receiving provider.
   	Open Rate	All channels and test components.	The “Unique opens” is the number of recipients who have opened the A/B test a unique number of times.
   	Click Rate	All channels and test components.	The “Unique clicks” is the number of recipients who have clicked the A/B test a unique number of times.
   	Unsubscribed	All channels and test components.	The percentage of recipients who clicked on the unsubscribe link.
   	Soft bounces	All channels and test components.	This is an email that couldn’t be delivered because of a temporary reason. Common reasons an email may soft bounce include: The mailbox is full. The message is too large. The server timed out.
   	Hard bounces	All channels and test components.	This is an email that couldn’t be delivered for a permanent reason. Common reasons an email may hard bounce: Fake email address. The email address no longer exists. Invalid email address. Email blocked by recipient’s server.
   	Marked as spam	All channels and test components.	Indicates the total number of users that have marked your Email Campaign as spam in their inbox.
   	Reject	All channels and test components.	The message was refused due to Smartsender.io account restrictions: The daily limit for sending emails has been exhausted. An attempt was made to send an email to a contact that is blacklisted. An anti-spam trigger has been triggered.

10. Press the “Info” button to get acquainted with the A/B Test details.

    Parameter	Description
    Channel	The type of channel.
    Status	The current status of the A/B Test.
    Failure reason	Template test failed	Invalid Template. This error message appears when some changes were made in the Template before the Test launching.
    	Not enough audience to test the variant	Less than 2 contacts (when 2 variants are testing) in a test environment. We recommend you have at least 5000 contacts for the A/B Test campaign.
    	Telegram Bot was deleted	Displays for two reasons: When the Telegram Bot was deleted. When the Telegram Bot is not available at the A/B Test start moment.
    	Email Template was deleted	Appears when the Email Template is deleted before the A/B Test starts.
    	WebPush Template was deleted	Appears when the WebPush Template is deleted before the A/B Test starts.
    	Telegram Template was deleted	Appears when the Telegram Template is deleted before the A/B Test starts.
    	Contact list was deleted	Appears when the Contact List is deleted before the A/B Test starts.
    	Filter variable does not exist	Appears when the variable in the selected/created filter has been deleted so then the number of contacts becomes not relevant, and the contacts do not correspond to the filter conditions.
    	Undefined telegram bot	Appears when either the token is out of date or has been deleted.
    	Invalid variants count	Not enough contacts have been assigned to test one of the variants. Contacts may have been deleted or they may have simply unsubscribed from the corresponding channel.
    Name	The name of the A/B Test campaign along with its ID.
    Creation date	The A/B Test creation date and time. Displays according to the account’s time zone.
    Template	The name of the Template along with its ID.
    Template A	The names of the Templates of the A/B Test campaign along with their ID.
    Template B
    Template C
    Email subject	The subject line of the email.
    Subject A	The subject lines of the email that undergo the A/B Testing.
    Subject B
    Subject C
    From name	The provided name that is displayed in your recipient’s inbox.
    From name A	The From names that undergo the A/B Testing. Your clients will receive emails from the provided names.
    From name B
    From name C
    TTL (Time to live)	The defined duration of the push service attempts to deliver the notification.
    Telegram bot	The name of the selected Telegram bot.
    Contact list	The name of the selected Contact List along with its ID.
    Launch type	The A/B Test launch type determines how the test is launched.
    Launch time	The A/B Test starting time.
    Launch time A	The A/B Test start time for each launch time testing variant.
    Launch time B
    Launch time C
    Criterion & Duration	Defined criteria for choosing the winning message and selected A/B Test duration. The winner will be determined based on the data for the specified time period.
    Tags	The list of Tags of the A/B Test.

11. By pressing the “More” button the following actions will be available:
    • Duplicate – create the new A/B Test with the same settings and parameters.
    • Delete – remove the A/B Test without a recovery option. NOTE The A/B Test could be also deleted by choosing the selection box next to the A/B Test row.
12. Edit the desired A/B Test by pressing on its name.