Sunday, November 20th, 2016
Sometimes you just can’t argue with the evidence. If a large sample of very ill people got better after dancing naked at full moon, then surely the dance works. Less contentiously, if the country’s best-performing schools produce worse results over time, then surely something is wrong with the education system.
But hang on a second. Before you jump to conclusions, you need to rule out a statistical phenomenon called regression to the mean. The idea is that if you choose a set of measurements because they are quite extreme, and then do the same measurements a little while later, the result is likely to be less extreme.
Regression to the mean is the statistical tendency of a data series to gravitate towards the center of a distribution, provided it starts on the either end of the distribution and is free to fluctuate. Specifically, it refers to the tendency of a random variable that is highly distinct from the norm to return to “normal.”
For example, if a researcher gave a large group of people a test of some sort and selected the top-performing 5%, these people would be likely to score worse, on average, if re-tested. Similarly, the bottom 5% would be likely to score better on a retest. In either case, the extremes of the distribution are likely to “regress to the mean” due to simple luck and natural random variation in the results.
Regression to the mean was first described by a cousin of Charles Darwing, Sir Francis Galton upon discovering that, on average, tall parents have children shorter than themselves and short parents have taller children than themselves.
Summing-up: Regression to the mean is an inescapable fact of life. It is one of the trickiest threats to validity. It is subtle in its effects, and even excellent researchers sometimes fail to catch a potential regression artifact.