Experimentation Works: The Surprising Power of Business Experiments

But none of the organizations described in this book started out as virtuosos. Everything they’ve accomplished has come through the careful design and redesign of experimentation systems and years of practice.

“For a vast digital product like the Journal, applying data-driven experimentation was like discovering plutonium; it’s the most powerful product development tool on the face of the planet. It allows us to safely test aggressive new changes, make laser-guided business decisions and to quickly, iteratively, improve our product.”

But there are several reasons why tests results don’t have to add up.

They want to see the return on investment (ROI) on experimentation before getting started, because that’s how they evaluate all new initiatives. In the past, I used to patiently explain the costs and benefits, so they could fill their spreadsheets for a financial analysis. But, as we’ve seen, the costs are tangible and the benefits are about opport

It’s easy to specify costs and risks up front; however, the benefits of action are often elusive, especially before the action is taken.

Becoming an experimentation organization will undoubtedly cause frictions, as for every action there will be an opposing reaction.

a business analytics program is incomplete without controlled experiments.

the jeopardy thesis is perhaps the most dangerous, because it asserts that a proposed action, though beneficial, involves unacceptable risk and costs.

it’s not about intuition versus experiments; in fact, the two need each other.