Store listing experiments that move the needle, and how to avoid the ones that lie.

A store listing experiment that declares a winner feels like proof. Sometimes it is. Sometimes it is a week of random variation with a green badge on it.

What you can test

Experiments are only for published apps, and they stop automatically after six months, at which point traffic returns to the current listing.

How Play decides a winner

• Target metric: unique users who click Install, Open or Pre-register. It counts users, not taps. This is the same click-based view as the store listing CTR Play now reports.
• Confidence level: how often the reported confidence interval will contain the true performance. Higher confidence means fewer false winners and longer tests.
• Minimum detectable effect: the smallest difference you care about. If the gap is smaller, the result is a draw.
• Results: Play says which option performs best, explains it with the interval and the effect size, or says "More data needed".

Habits that keep results honest

1. One asset per test. Google recommends it. Change the icon and screenshots together and you won't know which one worked.
2. Set the minimum detectable effect before you start. A 1% lift on a small app needs enormous traffic to detect; aim for a change big enough to matter.
3. Don't stop early. Peeking and stopping the day a variant looks good is the classic way to crown noise.
4. Cover full weeks. Weekday and weekend visitors behave differently.
5. Watch completion, not just clicks. A variant can win on Install clicks and lose on finished installs. Compare acquisitions after you apply it.
6. Re-check after applying. Compare the 28 days after with the 28 days before, on the same metric. Winners that don't hold were noise or seasonality.

What to test first

Start where the traffic already is: the first two screenshots and the icon, then the short description in a localized test. Our kids portfolio redesigned its screenshots five times based on conversion data before they settled. For Apple's equivalent, see product page optimization.

Questions people ask

How many variants can a Google Play store listing experiment have?

Up to two variants against the current listing. You can run one default graphics experiment or up to five localized experiments at a time.

What can I test in a Google Play experiment?

Default graphics experiments test the icon, feature graphic and screenshots. Localized experiments can also test the short and full description.

How long should a store listing experiment run?

Until Play reports a result at your chosen confidence level, covering at least full weeks. Play estimates the time needed during setup, and experiments stop automatically after six months.