Experimentation Works: The Surprising Power of Business Experiments

But none of the organizations described in this book started out as virtuosos. Everything they’ve accomplished has come through the careful design and redesign of experimentation systems and years of practice.

The problem, of course, was that it’s hard to find many princes if you’re kissing only ninety-seven frogs per year.

a reward system that punished failure, made their performance worse, leading to less experimentation—even

Consequently, somewhere around 50 percent of tests were winners—an unusually high success rate that should have set off alarm bells. Did the specialists run true experiments that result in gains and losses? Or did they verify something they already knew would work?

Having an online testing capability is an important competitive advantage, provided managers know how to use it.

Wandering is an essential counter-balance to efficiency. You need to employ both. The outsized discoveries—the ‘non-linear’ ones—are highly likely to require wandering.”

a business analytics program is incomplete without controlled experiments.

“blind” tests, which help prevent the so-called Hawthorne effect: the tendency of study participants to modify their behavior, consciously or subconsciously, when they are aware that they are part of an experiment

success “is not about episodic, momentary victories, though they do play a role. It is about the longer view of incremental steps that produce sustained progress.”