Quantifying the User Experience: Practical Statistics for User Research

Ideally, your sample is also selected randomly from the parent population. In practice this can be very difficult. Unless you force your users to participate in a study you will likely suffer from at least some form of nonrandomness. In usability studies and surveys, people decide to participate and this group can have different characteristics tha

One of the more famous examples of this distinction comes from the 1936 Literary Digest Presidential Poll. The magazine polled its readers on who they intended to vote for and received 2.4 million responses but incorrectly predicted the winner of the presidential election. The problem was not one of sample size but of representativeness.

The adjusted-Wald binomial confidence interval is one of the researcher’s most useful tools. Any measure that can be coded as binary can benefit from this confidence interval. In addition to a completion rate, another common measure of usability is the number of users likely to encounter a problem.

interactive lessons with many visualizations and examples on the www.measuringusability.com

Using the Empirical Rule and z-scores to find the percent of area only works when we know the population mean and standard deviation. We rarely do in applied research.

A simplified way of thinking about it is to think of the confidence interval as two margins of error around the mean. The margin of error is approximately two standard errors, and the standard error is how much we expect sample means to fluctuate given the sample size

There are generally two types of usability tests: finding and fixing usability problems (formative tests) and describing the usability of an application using metrics (summative tests). The terms formative and summative come from education (Scriven, 1967) where they are used in a similar way to describe tests of student learning (formative—providin

There are typically two types of summative tests: benchmark and comparative.

We can say there is a difference when one doesn’t really exist (called a Type I error), or we can conclude no difference exists when one in fact does exist (called a Type II error).