Multivariate testing
 | Multivariate testing, also known as MVT, or A/B/n
testing, is a campaign that allows you to send multiple message
templates (up to 8) to a part of the audience to compare their
effectiveness. After that, the winning template is selected and sent to
the rest of the audience. When the message is sent to each of the
recipients, the campaign will end in the same way as a
broadcast. To create a new multivariate test campaign, select Campaigns → Multivariate testing and press the + Create button. Also, you can use the campaigns calendar to create a new multivariate testing campaign. |
General settings
In general settings you can specify the campaign name, its description, tags to be associated with it and stats SubID. There is a toggle switch to activate the campaign in the right corner:
Campaign settings
The following information is defined in campaign settings:
- communication channels (Email, SMS or Push),
- subscribers,
- several templates with content,
- audience size for testing,
- selection of winner template.
Channels
On the first tab, you can select the communication channel you want to use. This channel must be available in at least one subscription resource. The resource is selected on the next tab.
For Email sending, you can use the field with the main subscriber contact information — Email field.
Subscribers
On the Subscribers tab you can choose a resource to send messages from and an audience you want to communicate to. It can be a database or a segment.
Campaigns are sent to customers who subscribe to a certain resource. Please, make sure that message recipients are subscribed to the resource before mailing.
In the Audience part for testing block, you set an audience part, that will receive several content kinds for comparing templates efficiency.
Templates
On this tab, you can select the content that you want to use in the campaign. In Altcraft, messages templates combine content for mailing by all available channels.
Please, make sure, that the template has content for the selected channel.
Press Add button 
and select the template. Any campaign type can use up to 8 templates. A/B testing campaign will send one message to each customer using selected templates in random order in equal parts.
Click on the template miniature to see a preview, or click on the template name to open the template editor:
Test settings
On this tab, you can configure how the winning template will be selected. The following options are available:
- Number of opened messages
- Number of message link clicks
- Select manually
Selection winner by openings/clicks
If you want to select the winning template automatically, set the campaign Start time. This is the time and date for the mailing test launch.
After launch, the campaign will automatically collect statistics by the total number of messages openings or link clicks. You can set a time for collecting statistics using Time to select the winner parameter.
After the campaign collects statistics on opens or clicks, the message template that is most effective based on the selected indicator will be sent to the main part of the audience according to the specified schedule:
The campaign schedule is described in detail on this page.
If you select a time zone other than your account zone in the schedule settings, only the winner template will be sent according to that time zone. Test messages will be sent according to the account's time zone.
To complete the setup, click Finish. The options you select will be displayed in the Campaign settings block. Using the button in the upper right corner of the block you can edit the settings:
Save the campaign to activate it.
Manual winner selection
If you want to use the Select manually option, set the Start time. Campaign start is the time and date of launch of the test part of the mailing.
Next, set the campaign schedule. According to this schedule, the winner template will be sent to the main part of the audience.
Save the campaign to activate it. After the campaign sends test messages, it will switch to Active (Waiting for you) status.
At this step, you need to analyze the performance of each template and choose the winner template. To do this, click on the "Make a decision" button in the campaign editing window:
You can also select the winning template in the window with the list of campaigns by clicking on the corresponding button in the context menu:
You will see a window with statistics for each template:
-
Send is the number of messages sent.
-
Delivery rate is the number of messages delivered.
-
Content CTR is the percentage of clicks in a message. Calculated using the formula: (Number of clicks / Number of messages delivered) * 100%.
-
Content OR is the percentage of openings in a a message. Calculated using the formula: (Number of messages opened / Number of messages delivered) * 100%.
If you don't have enough data to choose the winning template, you can use the Channel report. Click on the analytics icon to open it:
In the report, select the parameters necessary to evaluate performance. These can be absolute or relative values (such as Open Rate or CTR/CTOR). You can also check negative reactions by the number of complaints and unsubscriptions.
Instant campaign launch
If you want to launch a campaign right after its creation, toggle the appropriate option at the bottom part of the page.
If the limits were not configured, messages will be sent to all customers.
If limits were configured, the campaign will be launched besides schedule.
Communication policy
Communication policy sets limits for communication with one subscriber. It is configured in the Limitations block. By default, the campaign will use account policy or resource policy.
Smart settings
If Smart settings are activated for your Altcraft marketing platform account, you can manage campaign virtual senders, blind carbon copy settings and several other options. This section will appear at the bottom of the campaign editor page. The following settings are available:
Virtual sender — this option allows you to select a virtual sender for the campaign. The virtual sender is the set of rules for sending messages from real senders.
Custom tracking domain — this option allows you to track actions in campaign messages using additional tracking servers. Domains must be divided with line breaks.
Send hidden copy to these addresses — BCC for sent messages. Addresses must be divided with line breaks.
Extra options — these options depend on your Altcraft configuration. For example, these options can be Enable random tracking prefixes, or Do not check contact data for uniqueness.
If you are not sure, what Smart settings you need to use, please contact your Altcraft administrator or Altcraft support specialists.
Campaign watchers
You can set campaign watchers in smart settings. Watchers will receive the same messages as customers. Watchers database is necessary for this feature. It is created in the same way as a normal database, except special option which must be toggled in the general settings of the database.
In Smart settings, select Watchers database and limit:
