Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”

It is often easier to generate a plan, execute against it, and declare success, with the key metric being: “percent of plan delivered,” ignoring whether the feature has any positive impact to key metrics.

Revenue-per-user usually has high statistical variance, so it is not sensitive as a guardrail;

A simple hierarchy of evidence for assessing the quality of trial design

Controlled experiments are especially useful in combination with Agile software development (Martin 2008, K. S. Rubin 2012), Customer Development process (Blank 2005), and MVPs (Minimum Viable Products),

Slack’s Director of Product and Lifecycle tweeted that with all of Slack’s experience, only about 30% of monetization experiments show positive results; “if you are on an experiment-driven team, get used to, at best, 70% of your work being thrown away.

In general, for experimentation, you will be choosing the subset of business goal, driver, and organizational guardrail metrics that meet these measurability, computability, sensitivity, and timeliness characteristics.

Defining guardrail metrics for experiments is important for identifying what the organization is not willing to change, since a strategy also “requires you to make tradeoffs in competing – to choose what not to do”

when in doubt, measure more, but more importantly: think hard about what you are optimizing for.