Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
One rough rule of thumb is to try to limit your key metrics to five.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
For k = 5, you have a 23% probability of seeing something statistically significant. For k = 10, that probability rises to 40%.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Having too many metrics may cause cognitive overload and complexity,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
If you have multiple metrics, one possibility proposed by Roy (2001) is to normalize each metric to a predefined range, say 0–1, and assign each a weight. Your OEC is the weighted sum of the normalized metrics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
this type of review starts being effective in the late Walk or in the Run phases of maturity.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Using experiments as a guardrail is a difficult cultural change,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
when in doubt, measure more, but more importantly: think hard about what you are optimizing for.