Talk to us

Learn how you can Unlock Limitless Customer Lifetime Value with CleverTap’s All-in-One Customer Engagement Platform.

  • See a tailored demo of CleverTap's key capabilities.
  • Get a walk-through of industry specific use cases.
  • Obtain answers to any questions about integration, go-live, and support.

Your product demo

Get to know CleverTap from the scratch. To book your personal product demo, fill out the form and start at your preferred date. Afterwards we will get in touch with you.

Please enter a valid work email

Please enter a valid phone number

  • Talk to us

    • Advanced A/B Testing Calculator

    Step I: Message Variants

    This step involves selection of two or more message variants, say A and B, for testing. Usually, the first message is your default choice and the second message is competing to dislodge your default choice.

    Step II: Selection of User Groups

    This step involves choosing the right sample size for testing.

    Ideal Sample Size
    for each Variant

    – Conversion Rate The goal of A/B testing is to choose the message with the probability of having a higher conversion rate like CTR among the 2 messages. Though we do not know the conversion rate the campaign would ultimately clock, we need to make an educated guess based on the past conversion rates of similar campaigns.
    – Error While making an educated guess about the conversion rate, it is highly unlikely that you could confidently predict the exact conversion rate. In practice, it is always a range. For example: Instead of exact 3%, you might predict the conversion rate to be in the region of 2.7% to 3.3%. In other words, you have assumed an error of +/- 0.3%.
    – Confidence Level The confidence level lets you decide how sure you want to be about the sample results getting reproduced on the entire audience. If the actual CTRs for similar campaigns lie between 2.7% to 3.3%, a confidence level of 95% for the sample implies that 95% of the time, the actual CTR of the entire target audience would also fall in that range. Or put differently, you are willing to let your predictions about the CTR for the target audience fail only 5% of the time.

    Step III: Choosing the Winning Variant

    After you have selected the right sample size and ran the A/B test, this step involves evaluation of the tests results and selection of the winning variant, if any.

    Variant A Variant B
    Sample Size
    Converted
    Winner
    …
    Threshold Winner Confidence : 95%

    Now that you have understood how to choose your sample size, confidence level, error, and conversion rate; you can start testing various components of your message one by one. This will help you find an optimal message for your campaign. Check the CleverTap website to find more about A/B testing for push notifications.