How to Design Scientific Experiments That Help Mobile App Retention

Mobile apps are on the cutting edge of innovation.
Roughly 92% of young Americans (ages 18 – 29) now have smartphones in their hands, and companies like Uber, Snapchat, and Spotify owe their success to the power of mobile. They wouldn’t be here without that success.
So when mobile marketers attempt to grow an app, it’s tempting to lunge towards the newest and shiniest strategies.
But new isn’t always better. Way back in 1621, Francis Bacon formalized the scientific method with the publication of the Novum Organum Scientiarum — and it turns out that this method is the perfect way to improve user retention on mobile.
With advanced degrees in biology, Hila Qu, VP of Experiments & Retention at Acorns and former Product Manager of Growth at GrowthHackers.com, is well versed in this age-old strategy for running experiments and learning about the world. She employs the scientific method on a daily basis to find out what strategies can keep her company’s mobile users around.
Today, I’m sharing what Hila Qu and the scientific method can teach you about retaining more users on mobile.

A Quick Refresher on The Scientific Method

No matter your familiarity or experience with science, you likely learned about the scientific method in a high school classroom. The scientific method is an ongoing process for turning observations into provable facts.
It consists of making observations, formulating hypotheses, developing tests to determine whether hypotheses are accurate, and reporting on your results.

Image: Wikipedia

It’s easy to see how the scientific method could be applied to mobile marketing. In order to figure out which strategies will retain your mobile users, you have to make observations, formulate hypotheses, and test them out.
Qu has a bachelor and masters degrees in biology and conducted laboratory experiments where she mixed chemicals. “I still do experiments, except they are inside products or on different marketing channels,” she says. “Tweaking apps to experiment and make changes — this is very similar to scientific experiments.”

Why User Retention is the Perfect Playground for Experiments

Qu’s team at Acorns is primarily interested in user retention on mobile — and Qu is clear about her preference for working on retention, rather than acquisition.
“To a certain degree, new users can be bought by spending money on advertising, but you can’t use money to persuade users to stay,” she says. “User retention is the ultimate measure of whether everything — your mobile product, messaging, and overall experience — is working.”
[bctt tweet=”Retention is the ultimate measure of whether your product, messaging & overall experience is working —Hila Qu” via=”no” url=”http://bit.ly/2Aod768″]
In order to identify friction points and help mobile users get to “aha” moments more quickly, Qu and her team conduct in-product experiments, as well as experiments across retention channels like email and push notifications.
“We try to understand the entire user flow after acquisition, so that we can figure out the roadblocks users are facing,” says Qu. “We want to learn which barriers we can remove so that users can achieve their goals as quickly as possible.”
Qu and her team work to understand how users learn to navigate the app, when they get to “aha” moments that hook them on the app experience, and how to maintain regular engagement with features communications in the app, and on other channels, to keep the experience top of mind.

Mastering Mobile Engagement & Retention

Build campaigns that keep more of your app users from churning

Download Ebook Now

Applying The Scientific Method to User Retention

What Qu does at Acorns is very similar to what she used to do in a biology lab. Rather than testing her hypotheses on rats, Qu is running experiments to understand what gets users to stay.
Here are the steps Qu takes to apply the scientific method:

Step 1. Define the goal.

Qu starts with a goal, such as improving user retention for those who were acquired via a particular channel, or reducing barriers during the first three days.

Step 2. Generate hypotheses.

Once Qu has a goal, she can generate a set of hypotheses. These hypotheses are ideas for what the problems might be, and how making a change could improve retention.

Step 3. Design and conduct a test.

Perhaps the most interesting aspect of the scientific method is designing a test and conducting it. For example, if your hypothesis is that users are dropping off because they don’t understand a particular feature of an app, you might design a test where you offer a walkthrough of the feature to see if it makes a difference. Even for the same problem, there could be multiple solutions, so a lot of creativity and testing goes there as well.

Step 4. Analyze results.

Qu and her team analyze their results to determine whether a hypothesis is right or wrong. If it’s right, then the team is satisfied: they proved their goal and solved their problem. They make the change, apply whatever they learned to the next experiment, and look for any opportunity to further improve the results.
If the hypothesis is proven wrong, the team tries to understand if the result is due to a faulty experiment, or because the hypothesis itself was wrong. If the hypothesis is wrong, then that’s a conclusion that can be applied to later experiments as well.

Defining Performance Metrics That Make Sense for Your App

When it comes to running scientific experiments on your app, you have to define what healthy user retention looks like in your particular case.
“How you define retention is dependent on the behavior of your particular users,” says Qu. “A lot of companies will measure performance based on daily active users, but this might not be meaningful to you.”
For example, imagine you’ve created a note-taking app for college students. College students don’t have classes every day, so daily active users might not be a helpful metric.
For you, weekly active users might be a better metric. That means students might log in when they’re in class, when they’re completing an assignment, and when they’re studying for an exam. You will want to make sure you define “active” in a way that indicates they are really using the app, rather than just logging in.
Once you’ve settled on weekly active users, you can begin to draw hypotheses about what’s likely to positively influence the metric, then conduct an experiment to see if you’re right.

Considering Areas That Will Have the Biggest Impact on Your Metrics

When it comes to improving user retention, there are a thousand things you can do at any given time. How can you narrow down and choose to experiment on the areas that will have the biggest impact on your performance indicators?
You can create a mechanism to engage with current users, ideally turning your product usage into a habit. The other way is to encourage churned users to come back. You can also focus on educating new users so that they stick around.

Qu recommends creating a list of the different reasons why some stick around and why others leave. From that list, you can consider what the key difference is between these groups, form a lot of ideas, make an educated guess on what will move the needle, and then create an experiment.

“You can try different channels, different copy, different design, and you can appeal to different user psychology,” explains Qu. “Pick an area that you think has the biggest potential, and start from there. Take what you learn into the next experiment, then rinse and repeat.”

Hila Qu’s Top Mistakes to Avoid:

Hila Qu has been running experiments in mobile growth for years. Here are her top mistakes to avoid:

• Guessing instead of experimenting. “People still try to guess what is going to work best,” says Qu. “But it’s so easy to run a test on different marketing channels, or within your product.” Stop arguing with your team about what will work, and test things out. Sometimes the smallest change has the biggest impact, and the experiment you are so sure about ends up not working at all.
• Not taking a holistic view. According to Qu, improving user retention is a “full funnel exercise.” You must consider how you acquire new users, how you activate them, and how you can turn them into long-term users.
  After that, you have to consider how you can get them to not only stay but spread the word to their friends. You also have to consider what happens after a user has left: do you have a way to encourage them to come back?  
• Siloed teams that don’t collaborate. “Growth requires more and more core functional collaboration,” says Qu. “The simplest tasks, such as improving your ranking in the App Store, require the involvement of totally different teams.” If your team isn’t collaborating, then you’re going to have difficulty acquiring users and convincing them to stick around.

More About Hila Qu

Hila Qu is the VP of Experiments & Retentions at Acorns, an app that helps users save spare change, and was previously a Product Manager at GrowthHackers.com. She works with designers, engineers, marketers, and other stakeholders to launch experiments inside and outside the Acorn product to drive user activation, engagement, retention, and referral.