Worksheet 4 — Test brief

Use with chapter 4, Listing and conversion (also for paywall and pricing tests from chapter 5). Write it before the test starts. Do not edit the verdict rules after you have seen results.

App: ______________ Platform: ☐ iOS ☐ Android Market: ______________ Owner: ______________ Start date: __________

1. Message hypothesis

• Who (which visitors, from which source): ______________________________________
• What they want to know: ______________________________________
• What the page tells them now: ______________________________________
• What we will show instead: ______________________________________
• Is every claim true of the current version? ☐ yes ☐ no → stop

2. Test setup

• Tool: ☐ App Store product page optimization ☐ Play store listing experiment ☐ custom product page ☐ custom store listing ☐ other: ______
• Number of treatments/variants: ____ (fewer is better when traffic is small)
• Assets changed: ☐ icon ☐ screenshots ☐ preview/video ☐ feature graphic ☐ description text — ☐ tested as a bundle
• Share of traffic in the test: ____ % → each variant gets ____ %
• Needs App Review / new binary (e.g. alternate icons)? ☐ yes ☐ no

3. Can this test say anything? (planning estimate)

Assumptions: independent visitors; stable traffic and rate during the test; ~80% chance to detect a real difference of size d; 5% false-alarm rate; one variant vs original. This is a rough normal approximation for equal, independent arms; use an exact or store estimator for low rates, small counts, unequal allocation, repeated users or devices, or clustered traffic. Not the test's own statistics.

• Current rate p = ______ Difference to detect d = ______
• Visitors per arm ≈ 16 × p × (1 − p) ÷ d² = ______
• Visitors per arm per week: ______ → weeks needed ≈ ______
• Tool time limit (App Store PPO: 90 days; Play experiments: 6 months): ______
• ☐ Fits ☐ Doesn't fit → make the difference bigger, send more traffic, use the largest market, or record as a judgment call

4. Metric and stop rule

• Primary metric (exact definition): ______________________
• Guardrail metric (must not get worse, e.g. trials per download): ______________________
• Minimum run (planned sample per arm, rounded up to full weeks): ______ weeks — no decision before this, whatever the tool shows
• Maximum run (tool limit or my own, whichever is shorter): ______
• After the minimum run, stop when: ☐ the tool reports a result at the confidence I set (App Store: Apple suggests at least 90%) ☐ the maximum run is reached → record the estimate and interval or precision information available; do not infer a small effect from a nonsignificant result, and mark inconclusive unless the planned precision excludes a meaningful effect
• Frozen during the test: other listing changes in this market ☐, price ☐, releases touching onboarding ☐

5. Pre-committed actions

6. After the test

Result: ______________________ Recorded in decision journal on __________ ☐