Industry Best Practices & Top Insights delivered to your Inbox.
Blog Product

Introducing Control Groups: Measure Actual Marketing Efficacy

Subharun Mukherjee 18+ years of experience leading product strategy, Go-To-Market (GTM), new market entry, value-based sales, analyst relations, and customer experience programs. Expertise in Financial Services, eCommerce, on-demand services, and the SaaS industry.
Introducing Control Groups: Measure Actual Marketing Efficacy

If marketing campaigns had existential crisis, they would seek answers to questions like:

  • Why are we here?
  • Would anyone notice if we suddenly stopped showing up?
  • What impact did we make?

Although hypothetical, the questions are pertinent. As marketers, you need to be aware of how your campaigns are performing. Whether the campaigns are actually moving the needle or if there’s something else influencing conversions.
Theoretically, you would want to create a campaign, send it to your user base, measure the performance, then go back in time and choose not to send the same campaign and compare the performance. The difference in the performance between these two scenarios is the real impact of your campaign.
But since this is neither efficient nor doable, the only way to understand the actual impact of your campaign is to isolate users from receiving the campaign using a control group then measure campaign effectiveness by comparing this group with the Target Group that receives your campaigns.

Measuring the Effectiveness of your Campaigns

Given the vast difference in behavior of your users, it becomes increasingly important to know the impact of your campaigns across the entire spectrum of your user base.
Consider a promotional campaign for Loyal Customers meant to drive repeat purchases in the holiday season. The initial result of the campaign can lead you to believe that an increase in purchases is due to the campaign. But what if these users were already on the path to purchase?
You can now compare a Control Group’s activity to the Target Group’s activity. Thereby understanding the real impact your campaigns make.
Measuring the real impact of your campaigns also helps in tying marketing dollars spent to dollars earned. You can easily get insights like:

  • Average Revenue Per User (ARPU): The ARPU for users who received engagement campaigns over a time period is more than ARPU for users who didn’t.
  • LTV: The LTV of users in the Target Group is greater than the LTV of users in the Control Group, who didn’t receive any campaigns.
  • Uninstall rates: Uninstall rates amongst users receiving push messages (Target Group) are higher than the ones not receiving them (Control Group). Meaning, sending too many engagement campaigns is possibly leading to app uninstalls.

Such impactful insights are incredibly valuable and can be easily derived with CleverTap’s latest feature: Control Groups.

CleverTap Control Group

Control Groups at a Glance

CleverTap allows you to set System Control Group and Custom Control Groups.

  1. System Control Group (SCG): Used to measure the impact of engagement campaigns over the long-term (over months and years). SCG is an account level control group which is applicable to all campaigns. For cases like sending transactional messages or app update reminders that need to go to all users, you may disable SCG.You can define the control group limit (say 2%), and the system qualifies the specified percentage of users in the Control Group in a completely random way.

    CleverTap System Control Group
  2. Custom Control Group (CCG): Used to measure the impact of any single campaign or group of campaigns over a relatively short period (e.g. performance of engagement during the Superbowl). You can define a CCG the same way as a System Control Group.
    CleverTap Custom Control Group

Control Group Pros and Cons?

Control Groups are sometimes viewed with a healthy skepticism. Some marketers are reluctant to use it at all because it means some users miss out on marketing messages. Unfortunately, they don’t realize that they can always send the campaign to the control group after the test is over.
There’s tremendous upside to using control groups and measuring the actual performance of your marketing campaigns. It is better to test and send the right campaign to some of your user base than to send the wrong one to all at once.

Final Thoughts

While A/B testing tells you which of two (or more) options you should pick, it doesn’t tell you anything about the scenario if you had not nudged users at all. Through Control Groups, you can find out whether your campaign might perform better or worse than those who did not receive the campaign. In case it’s the latter, Control Groups can tell you if you should refrain from sending the campaign at all and simply let users follow their path naturally.
Control Groups gives you the power to measure the actual performance of your campaigns. You can learn more about it in our product documentation or write to us at

Last updated on March 5, 2024