3 MINUTES BULLSHIT WITH GEORGE #44: Subscription A/B Testing Pitfalls with David Barnard

Episode #44 of 3 MINUTES BULLSHIT WITH GEORGE featured David Barnard, Growth Advocate at RevenueCat, Host of the Sub Club podcast, and Founder at Contrast. We discussed how to avoid common pitfalls in subscription price A/B testing.

You can listen to the full episode here:

In this episode of 3 MINUTES BULLSHIT WITH GEORGE, George Natsvlishvili is joined by David Barnard, Growth Advocate at RevenueCat, Host of the Sub Club podcast, and Founder at Contrast. RevenueCat is an in-App purchase and subscription platform for mobile apps. Sub Club podcats is a show about building and growing app businesses. Contrast is an apps studio with several apps.

David shares his experience as a Mobile Growth Expert in the subscription price A/B testing field.

Check out all the other episodes of 3 MINUTES BULLSHIT WITH GEORGE on our website or on Youtube channel.

Connect with David and RevenueCat here:

Timestamps:

00:00 Dance

00:14 David’s Introduction

00:29 Subscription A/B Testing Pitfalls

04:59 Bloopers

Transcript & Key takeaways from the episode:

Here are tips to avoid common pitfalls in subscription price A/B testing:

Avoid Short-Term Bias: Initial A/B test results can be misleading, especially in subscription apps where long-term retention is key.
Monitor Renewals: Higher prices may increase short-term revenue but could reduce renewal rates, hurting long-term growth.
Track Long-Term Cohorts: Measure KPIs beyond 7 days, ideally for months or even a year, to assess real impact.
Retention Over Revenue: Long-term retention is the ultimate KPI—sustained subscriptions indicate real value.
Beware of Over-Optimizing: Short-term gains in conversion rates may sacrifice long-term sustainability.
Experiment Tracking Matters: Utilize tools that monitor results over extended periods to avoid premature conclusions.

George: How to avoid getting fooled during A/B testing and which KPI’s should you focus on?

David: Yeah, so I focus a lot on subscription apps. I’m working at RevenueCat and I run subscription apps for myself. And I've been doing A/B testing on my own app recently. I've talked on the podcast, the Sub Club podcast that I run I talk about testing and you know one of the things that I've talked a lot about and the people have had on the podcast talked a lot about is that it is really easy to get fooled by the initial A/B test results and especially with subscription apps where to a build a really great business over the long haul you really need that long-term retention. With a lot of pricing experiments you're gonna make more money increasing your price. I'm just seeing over and over again. Talk to different founders increasing the price generally leads to more revenue, so the initial A/B test will show increased revenue. Not always the case but often. But that's a good example that sometimes doing that test even if it's a clear winner we go all the way around to the renewals of those annual plans: that higher price might renew it at a much lower rate than the lower price. And so if you're trying to build a long-term business, and you're doing A/B testing where you are only looking at seven days of results or you're only tracking the AB you're only tracking that cohort for days, or even a month. You know some people don't even track this cohort's past the data that collect during the A/B tests. And making those kinds of short-term decisions on what you should be focusing on this long-term retention. You can really be shooting yourself in the front bringing in the wrong cohorts, pricing too high for retention, and just doing a lot of short-term optimizations that are undercutting the long-term strength and growth of your business. And so as far as KPIs to measure retention is kind of the ultimate KPI for subscription app business. If you're delivering value to customers, they will stay subscribed. And if you're delivering value enough, so that a lot of people are staying subscribed then that probably also means that you're gonna have an easier time converting them to subscription through ads and ASO and other stuff, so yeah. There are a million different KPI’s to track along the way. You need to optimize the type of funnel event, but you want to make sure you're not over-optimizing the top of the funnel for short-term gains after sacrificing of the long-term growth and sustainability your business.

George: Got it. So you have to follow the longer cohort than 7 days during A/B test to make sure that this is the long-term bet and not to short-term win.

David: Exactly. I ran an A/B test in my own weather app using RevenueCat’s tools and I ran it for I think a week. And I turned off the A/B test. But the way our experiments feature works is actually tracks for 400 days. And so initially I had way higher realized LTV in the control group than the experiment group. But the experiment group is tacking up because it's a monthly subscribers: keeping in, keeping in, keeping doing. And initially, I just thought it was like okay clearly this is a winner, but like now I'm less and less confident that it was the right decision because those renewals are coming in at a much higher rate than I thought and I'm watching this cohort over a long period of time to see if my initial thoughts were actually correct. And but then there are all the other factors that come into play: annual subscriptions vs. monthly and you get more money from with the annual and this was pitting monthly against annual. So we'll see. But that's the point. It's like we'll see. The test was run a month ago, and I've been tracking it in every sense. I'm going to keep tracking it to make sure that it is actually what's going to produce the best outcome for my business in the long run.

► Subscribe to check the other episodes.

If you have any questions related to online marketing leave your comments/questions below. I will find the expert to answer your question.