Regression to the mean describes the feature that “extreme” outcomes tend to be followed by more “normal” ones.
When we witness “extreme” events such as unlikely successes or failures, we forget how rare such events are. When these events are followed by more “normal” events, we try to explain why these “normal” events happened .
This often leads us to attribute causal powers to people, events, and interventions that may have played no role in bringing about that “normal” event.
[This article was originally published on Medium by Toward Data Science]
Here are some things I have either read, heard, or said in the past. Can you spot anything wrong with these statements?
- A teacher laments: “When I praise my students for good work, the next time they try, they tend to be less good. When I punish my students for producing bad work, the next time they try, they tend to do much better. Therefore, punishments work but rewards do not.”
- An aspiring athlete marvels: “Wow, yesterday my foot was really painful and I soaked it in a hot bowl of garlic-infused water, now it feels much better. There must be some healing properties in that garlicky water!”
- A film critic stipulates: “Winning an Academy Award, the most prestigious prize in show business, can lead an actor’s career to decline rather than ascend.”
Your hunch might be that there’s something fishy with these statements. Surely, garlicky water can’t cure a painful foot! This article explains why these statements are problematic through the concept of regression to the mean and suggests some steps you can take to avoid making this fallacy.
Regression to the Mean
Let’s take a look at the pattern of events spoken about in these statements. In all these statements, events unfold as follows: an “extreme” event (either in a good or bad direction) is followed by a more typical one.
We often observe events unfolding in this pattern (extreme, typical, typical, extreme…) because of a statistical phenomenon called regression to the mean or reversion to mediocrity. Regression to the mean refers to the idea that rare or extreme events are likely to be followed by more typical ones. Over time, outcomes “regress” to the average or “mean”.
The term was coined by Sir Francis Galton when he noticed that tall parents tend to have children shorter than them, whereas short parents often have children who were taller than them. Graphically, this means that if we plot the height of parents on the x-axis and the height of kids on the y-axis and draw a line through all the data points, we get a line with a slope of less than 1 (or equivalently an angle of less than 45 degrees).
This diagram shows that a father who is taller than the average man in the population will, on average, have a son that is slightly shorter than him. A father who is shorter than the average man in the population will, on average, have a son that is slightly taller than him.
This concept is useful not just for looking at the relationship between the heights of parents and their offspring. If you reflect on your own experience, you’ll most likely find this phenomenon popping up everywhere. For example, regression to the mean explains why the second time you visit a restaurant you thought so highly of last time fails to live up to your expectations. Surely, it can’t just be you who thinks the creme brulee was better last time!
The statistical feature that explains such a phenomenon is that getting multiple extreme outcomes consecutively is like getting heads on a coin multiple times — unusual and difficult to repeat. The various factors that make a restaurant experience optimal — the quality of the food, how busy the restaurant is at the time of your visit, the friendliness of the waiter, your mood that day — are difficult to repeat exactly the second time around.
The general statistical rule is that whenever the correlation between two variables is imperfect (i.e. in the graph above, the slope is not 1), there will be regression to the mean. But… so what?
Regression to the mean is a statistical fact about the world that is both easy to understand and easy to forget. Because the sequence of events unfolds in this way (extreme, typical, typical, extreme…), our brains automatically draw some relationship between the “extreme” event and the “typical” event. We (erroneously) infer that the extreme event caused the typical event (e.g. the Oscar win led to subsequent mediocre movies) or that something in between those two events caused the typical event (e.g. the garlicky water made the foot feel “normal”).
In short, when we forget the role of regression to the mean, we end up attributing certain outcomes to particular causes (people, prior events, interventions) when in fact these outcomes are most likely due to chance. In other words, the “normal” outcome would have occurred even if we removed the prior “extreme” outcomes that came before it.
Let’s cast our minds back to the first 3 statements in this article. What does regression to the mean imply about the veracity of these statements?
The teacher’s mistake:
The teacher is suggesting that rewarding his/her students makes them perform worse next time. But seen from the framework of regression to the mean, the teacher is claiming too much credit for her reward/punishment system.
As Daniel Kahneman notes in Thinking, Fast and Slow, Success = Talent + Luck. The “luck” component of this equation implies that a string of successes (or failures) are difficult to repeat, even if one has lots of talent (or no talent).
The “luck” component of this equation implies that a string of successes (or failures) are difficult to repeat, even if one has lots of talent (or no talent).
Consider two students Jane and Joe. In year 1, Jane does horribly but Joe is outstanding. Jane is ranked at the bottom 1% while Joe is ranked at the top 99%. If their results were entirely due to talent, there would be no regression — Jane should be as bad in year 2 and Joe should be as good in year 2 (possibility A in the diagram below). If their results were equal parts luck and talent, we would expect halfway regression: Jane should rise to around 25% and Joe should fall to around 75% (possibility B). If their results were caused entirely by luck (e.g. flipping a coin), then in year 2 we would expect both Jane and Joe to regress all the way back to 50% (possibility C).
The Success = Talent + Luck equation means that the most likely scenario is some degree of regression. Jane who did awfully in the first year is likely to do somewhat better in the second year (regardless of whether she is punished) and Joe who did extremely well in the first year is likely to do somewhat worse in year 2 (regardless of whether he is rewarded). The teacher’s punishment and reward system may not be as powerful as she thinks.
The athlete’s mistake:
The athlete is suggesting that the garlicky water healed her painful foot. Suppose that the athlete’s foot was hurting really badly one day. In desperation, she tries everything, including the garlic-infused water right before bed. She wakes up and finds that her foot feels a lot better. Perhaps the water does have some healing properties. But regression to the mean seems to be an equally plausible story.
Since physical maladies have a natural ebb and flow, a really painful day is likely to be followed by a less painful day. The pattern of how pain rises and falls might make it look as though the garlicky-water is an effective treatment. But it’s more likely that the garlicky-water didn’t do anything. The pain would be subsided anyway due to the natural course of biology.
This example shows how regression to the mean can often be used to make ineffective treatments look effective. A quack who sees their patient at their lowest point might offer an ineffective medicine, betting that their patient will be less sick the next day and infer that the medicine is what did the trick.
The film critic’s mistake:
The film critic is alluding to the Oscar curse, which says that once you’ve won an Oscar, your next movie and performance aren’t going to be as good (it might even be bad). Several interviews with actors and actresses suggest that they also believe in the curse, saying that the pressure and expectation that comes with winning such a prestigious award ends up hurting their careers.
While pressure and expectations might explain the success (or lack thereof) of their next film, the “curse” could also be in large part due to regression to the mean. Actors sometimes have really good performances — a combination of a good screenplay, a good director, good chemistry with other actors, and a right fit for the part. Statistically speaking, the odds of all these factors coming together in the same dose and in the same way for their next movie project are low. So, it seems natural to expect their next performance to be worse than their Oscar-winning performance. The “curse” isn’t so mystical after all.
Regression to the mean describes the feature that “extreme” outcomes tend to be followed by more “normal” ones. It’s a statistical concept that is both easy to understand and easy to forget. When we witness “extreme” events such as unlikely successes or failures, we forget how rare such events are. When these events are followed by more “normal” events, we try to explain why these “normal” events happened — we forget that these “normal” events are…normal and that we should expect them to happen. This often leads us to attribute causal powers to people, events, and interventions that may have played no role in bringing about that “normal” event.
To avoid making this fallacy, we should first catch ourselves when we’re trying to explain some (either positive or negative) event or outcome. Then, we should ask ourselves the following questions:
- Is there any “abnormal” about this outcome or is this what I should expect, statistically speaking?
- Was this event preceded by some “extreme” outcome that makes the “normal” one look “strange” in comparison?
- Would the “normal” event have happened anyway, even if we removed the events before it?” For example, would the athlete’s foot have felt better even if she didn’t soak it in garlicky water?
Question 1 forces us to consider the likelihood of the normal event happening. Question 2 encourages us to think about outcomes in relation to each other, not as isolated observations. Question 3 pushes us to engage in counterfactual thinking, imagining a world where the entity that we believe to have caused the “normal” event has been removed. By asking ourselves these three questions, we are less likely to ascribe unwarranted powers to events, people, systems, and interventions.