Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
If you have multiple metrics, one possibility proposed by Roy (2001) is to normalize each metric to a predefined range, say 0–1, and assign each a weight. Your OEC is the weighted sum of the normalized metrics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
always having a portfolio of ideas: most should be investments in attempting to optimize “near” the current location, but a few radical ideas should be tried to see whether those jumps lead to a bigger hill.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
EVI: Expected Value of Information from Douglas Hubbard (2014), which captures how additional information can help you in decision making. The ability to run controlled experiments allows you to significantly reduce uncertainty by trying a Minimum Viable Product (Ries 2011), gathering data, and iterating.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Slack’s Director of Product and Lifecycle tweeted that with all of Slack’s experience, only about 30% of monetization experiments show positive results; “if you are on an experiment-driven team, get used to, at best, 70% of your work being thrown away.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Having too many metrics may cause cognitive overload and complexity,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
When you have an online business, you will have several key goal and driver metrics, typically measuring user engagement (e.g., active days, sessions-per-user, clicks- per-user) and monetary value (e.g., revenue-per-user). There is usually no simple single metric to optimize for.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
It is often easier to generate a plan, execute against it, and declare success, with the key metric being: “percent of plan delivered,” ignoring whether the feature has any positive impact to key metrics.