Roman's Data Science: How to monetize your data

The null hypothesis is rejected if your data set is unlikely to have been produced by chance. The significance of the results is described by the confidence level that was defined by the test (as described by the acceptable error “alpha-level”).

A/B and multivariate tools identify when treatments have reached a level of significance where we can make decisions. For example, a simple computation might show that version A has x participants of which y percent converted and version B has m participants of which n percent converted. You

Power analysis helps you manage an essential tradeoff. As you increase the sample size, the hypothesis test gains a greater ability to detect small effects. This situation sounds fantastic. However, larger sample sizes cost more money. And, there is a point where an effect becomes so miniscule that it is meaningless in a practical sense.

Whereas statistics are concerned with probabilities and confidence, business is about making good decisions. The most accurate conversion rate you have at any time is the one you have observed. In other words, the “real” conversion-rate lift is the observed conversion-rate lift we’ve tracked in the test: