Introducing A/B & Multivariate testing plus more

Optimizing messages and campaigns is a crucial activity for any app marketer. We’ve just added three new ways to optimize everything you deliver –

• A/B & Multivariate Testing
• Split Message Delivery and
• Sending campaigns to an Audience Subset

All three features are now available for Push. The other marketing channels we’ll roll out in the next few weeks.

A/B Testing

A/B Testing lets you compare up to three different versions of a message for any campaign. You can try different copy, creatives, calls to action or any combination to make sure you’ve designed the best message for each campaign. You pick the size of the test audience to run and we’ll do the rest.
We’ll determine the winner based on the number of click-throughs, and then deliver the winning message to the rest of your target audience. Simple!

Here’s how it works:
For any campaign you’ll need the content and creatives for each message. When creating a Push Notification campaign you’ll now see an option to add up to three variants of any message to test which one performs best.

After you’ve added the desired number of variants you’re ready to determine how big of a test to run.

Campaigns sent to Past Behavior Segments

For campaigns sent to Past Behavior Segments (grouping of users based on what they have done in the past) you have two options – launch the A/B test to a percentage of your target audience or send out an absolute number of messages. In either case we’ll deliver the variants equally to the test audience.


For example:

• if you are testing three messages (Variant A, Variant B, Variant C)
• and your Campaign Reach is 2,000,000 users
• and you choose a Test Population of 15% of Campaign Reach (300,000 users)

then we will send:

• Variant A to 100,000 users
• Variant B to 100,000 users
• Variant C to 100,000 users

After all 300,000 messages have been delivered, we’ll calculate the winning message over this test group based on the number of click-throughs. We’ll then automatically send the winning message to the remainder of your target audience – 1,700,000 users in this example.

Note that for A/B testing, we ensure there is always an equal number of messages sent for each variant. This ensures there is no bias introduced during the test phase and that the best performing message is always declared the winner.

Live User Segment (Triggered) Campaigns

With campaigns sent to Live User Segments (triggered) campaigns messages are delivered immediately when a user’s activity matches the criteria you’ve selected. For example: send a message when the user has completed a booking or purchase. Since it’s not possible to determine the reach of triggered campaigns up front you’ll need to decide how many total messages to send for A/B testing before a winner is declared.
For example, if you select 500 users as your test audience. We’ll alternate delivery of Variant A and Variant B as users qualify for the campaign. After 500 total messages are sent (Variant A – 250 and Variant B – 250) we’ll decide the “Winner” based on the number of clicks and continue only with this winning message for the duration of the campaign. Don’t you love us already 🙂
Deciding on a test audience for A/B Testing triggered campaigns requires some estimation. We recommend you check the total messages that were sent for similar triggered campaigns in the past to get a sense for how many users may qualify. If you select a test audience that’s too small (say 25 users) you’ll get a statistically insignificant sample. If your test group size exceeds the total number of users who ultimately qualify for that campaign, then no “Winner” will be declared and each message variant will be alternatively delivered for the duration of the campaign.

Split Delivery

For some campaigns you want to send multiple variants of a message to your entire target audience without picking a winner. With Split Delivery you choose what percentage of your target audience receives each variant and we’ll deliver them accordingly.

Here’s how it works:
As with A/B testing, when creating the message for a campaign you’ll have the option to add up to three variants. With Split Delivery you can decide what percentage of your audience receives each message variant. We’ll deliver the messages in exactly this proportion for the duration of the campaign.

Once the campaign is over, go to the campaign stats to compare how each message performed.

Deliver to a Subset of Target Audience

Sometimes you want to send a message to only a subset of the qualifying audience (Target Reach) for a campaign. A great use case for this is, say a limited offer you wish to send – a fixed number of coupon codes you want to distribute. If the total reach for your campaign exceeds the number of coupon codes you can distribute you can simply limit the number of users who will receive the message to exactly the number of coupons you wish to distribute.

Here’s how it works:
In the campaign creation flow, when you are determining who to send your message to, under the Campaign Reach section, select the option to limit delivery of the messages to the number you choose.

For triggered campaigns (based on Live User Segments), as users qualify, they will receive the message until the total quantity specified has been delivered, after which the campaign ends. For campaigns based on Past Behavior Segments we’ll randomly select the users that receive the message.