Custom product page tests on App Store and Play Console are valuable because the store is already delivering intent traffic — the user searched for something, you showed them a page, and you can measure the install rate on the page you actually shipped. The reason most tests under-deliver is not the tooling, it is the test design: too many variables moving at once, a sample size too small to read, and a campaign budget that has not been trimmed in the middle.
Pick exactly one of three variables per test: the first screenshot (the visual that earns the scroll-stop), the subtitle (the line of text under the title), or the call-to-action phrasing inside the first screenshot caption. These three cover the vast majority of the install-rate elasticity you have control over; everything else is a re-arrangement. Two variables is a coin flip; three is a fortune telling session. The same three variables are the ones /dashboard/ab-tests rotates on for our Full Launch tier — pick one, draft the variant, the agent ships it.
Set a minimum detectable effect of 10 percent relative lift before you start the test, not after. With App Store install conversion rates in the 25–40 percent range, a 10 percent relative move is the floor where the test result survives the noise of organic vs paid traffic asymmetry. Below that, you will get a winner that flips on retest, every single time.
Run for a fixed calendar window — fourteen days, not 'until the result is significant'. Significance-driven peeking inflates your false-positive rate by a factor that compounds fast. Fourteen days also covers the full weekly seasonality most consumer apps have, so you are reading an honest average rather than a Wednesday effect pretending to be a permanent lift.
Finally, when traffic is asymmetric (one variant gets most of the impressions because of budget allocation differences), do not trust the install-rate lift directly. Normalize by impression volume and read cost-per-install instead — it absorbs the asymmetry and tells you which variant actually earned its spend.
A/B testing sits in Phase 03 of the free Solo-Founder App Store Launch Checklist — the PDF lays out which variable to pick, the fourteen-day window, and the cost-per-install read as one ordered check.