Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
For education, establishing just-in-time processes during experiment design and experiment analysis can really up-level an organization.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
when in doubt, measure more, but more importantly: think hard about what you are optimizing for.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organization may reject new knowledge that is contradictory per the Semmelweis Reflex
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
always having a portfolio of ideas: most should be investments in attempting to optimize “near” the current location, but a few radical ideas should be tried to see whether those jumps lead to a bigger hill.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Make it hard for experimenters to launch a Treatment if it impacts important metrics negatively.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve (Kohavi, Crook and Longbotham 2009). Success is even harder to find in well-optimized domains like Bing and Google, whereby some measures’ success rate is about 10–20% (Manzi 2012).
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Using experiments as a guardrail is a difficult cultural change,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
hold classes to keep employees aware of experimental concepts