Shocking Results Need Mundane Assumptions
The rule for persuasive analyses
All analyses rely on assumptions to map datasets to conclusions. The data doesn’t speak for itself. You have to shape it. Ideally, the mold you fit the data into is based on a reasonable theory of how the world works.
Analysis is a trick to, with enough data, reduce the world to two potential outcomes:
The analysis’s conclusion is true.
OR: the assumptions of the analysis are wrong.
This is very useful. It moves the conversation forward. Folks can’t just disagree with your conclusions. They have to disagree with your assumptions, too.
Therefore, an analysis is persuasive only if your assumptions are more plausible than your conclusions.
If your assumptions are more plausible than your conclusions, then disagreeing with the conclusion becomes more difficult after the analysis. Before I could argue against the conclusion directly. Now, I have to argue against the assumptions, which are harder to dispute.
But if your assumptions are less plausible than your conclusions, then you haven’t convinced me. After the analysis, I know either the conclusion is true, or your assumptions are wrong. If I disagreed with the conclusion before, and it is even easier for me to disagree with your assumptions, it is just as easy to disagree with you as it was before your analysis*.
There’s a famous paper in empirical labor economics that illustrates this.
Card and Krueger (1994) claim to show no effect of a minimum wage increase on unemployment.
They do so using a difference-in-differences analysis of bordering counties in Pennsylvania and New Jersey after New Jersey raised its minimum wage. The idea is that these counties are similar to each other and face similar shocks, except that the New Jersey counties received a higher minimum wage. Card and Krueger look at changes in employment and find a small, positive, but insignificant effect. The conclusion is that the minimum wage increase did not cause unemployment. Politicians frequently cite the paper to draw the much broader conclusion that minimum wage increases don’t, more generally, cause unemployment.
It isn’t a persuasive analysis. It’s further confirmation to the believer, but it won’t convert anyone.
Suppose that, prior to the analysis, I disagree with the idea that labor demand is flat or upward-sloping. Now, after the analysis, you tell me: either you have to believe that demand is flat or upward-sloping, or you can believe that these New Jersey and Pennsylvania counties are not really comparable.
Of course, I’m going to conclude that these counties aren’t really comparable. If I have to choose between “People don’t care how much they pay for things” and “These counties aren’t comparable in some subtle way. Maybe another policy or shock that happened at around the same time wasn’t distributed evenly across New Jersey and Pennsylvania counties. Maybe these counties are just wholly incomparable because of a long history of differences in policy and polity between the two states that change how they respond to various shocks. Maybe a silly shock that we’ll never understand happened. Maybe the unemployment effect is delayed because of adjustment frictions.” All of those explanations are more credible and convincing to me than “people don’t care about how much they pay for things.”
So, the study fails to convince anyone. Proponents of a higher minimum wage like the study, but they don’t need to be convinced by it. Sometimes academic economists who find themselves in policy roles where they need to pretend to like minimum wages might find it useful as air cover to explain away past writing that says minimum wages cause unemployment. “Oh, I used to think minimum wages were bad, but then I read Card and Kreuger.” Hot take: No true believer in downward-sloping demand curves has ever been convinced by this. It’s so much easier to conclude that parallel trends don’t really hold or the post period isn’t long enough to see the effect than to invalidate a simple, powerful economic theory on the basis of a diff-in-diff.
This isn’t to argue about an economics paper from more than thirty years ago. It’s to demonstrate that:
Shocking results need mundane assumptions to convert new folks to your way of thinking.
We should be credibility-maxing?
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn
*This point isn’t about Bayes’ rule. I understand how my joint prior over the assumptions and conclusions might cause my posterior beliefs about the conclusion to change, even if I view the assumption as less credible than the conclusion in terms of my prior’s marginal distributions. It’s an argument about how people are actually convinced, i.e., I don’t have a tightly constructed joint prior distribution over the likelihood of New Jersey and Pennsylvania counties being comparable and the slope of the labor demand curve. I weigh how plausible the assumptions are and how plausible I think the conclusion is, and that determines whether I end up convinced by the analysis to change my mind.