“I just need a number.”
And it’s true. We’ve got a question to answer and a call to make. An analysis littered with careful, intellectually humble caveats might sound like a good idea, but it obscures that reality. We have to pretend some number is the truth, and act on it.
If you give the business an analysis sans a bottom-line number, then someone else will choose the number. There’s no avoiding it. Demand begets supply.
In causal inference, we talk about the “endogeneity problem”, unobserved factors driving selection into treatment. It’s one of those things you can always say in a seminar to pretend you’re paying attention. “I’m not convinced rainfall is endogenous…”
In industry, we have an “exogeneity problem.” The questions we have to answer and the numbers we need to make a decision are exogenous. You have to answer the questions you are asked. You can’t choose your paper’s subject. Little things like “we don’t have the data to answer that” don’t let you off the hook.
This environment drives the popularity of the difference-in-differences method—at least, I hope it’s not because people like making cardinal assumptions about relative slopes.
It’s popular because it’s possible. You nearly always have the data required to run the analysis:
Treated people.
Untreated people.
A time when the treated people weren’t treated.
These three little things let you run a causal analysis without introspection. No need to construct an instrument or model. You get a number in the end, and it’s easy to explain:
“Before we launched The Cool Thing, the difference between these two groups was 5. After we launched it, the difference grew to 10.”
[See how slick I am, weaving causal words like “grew” into what sounds like a mere factual summary? That is a trick you can only learn with experience.]
Parallel trends is a really peculiar assumption. It doesn’t hold under increasing transformations of the outcome, and depends delicately on a very specific functional form for the trend.
I’ve never been a big fan, but I’ve done it before. Recently, even. I didn’t inhale.
I proposed an ordinal version of the parallel trends assumption once, but it loses a lot of the key advantages that make difference-in-differences popular. You often can’t say anything at all with partial identification via monotone differences. It’s too cautious, too academic.
I’ll try to do better this time…
The thing is that the convenience of difference-in-differences is a real advantage. Outside of the academic world, among us mere industry hacks, endogeneity problems are much less salient than the exogeneity problem.
Saying, “well, the data doesn’t really tell us,” might be the most intellectually robust, humble, maybe even honest answer, but it’s usually a bad answer consequentially. The question will be answered by someone, some way, somehow. The business doesn’t have the option to not have an opinion on whether the new initiative worked. They have to decide whether to kill it or invest in it. So, you do the best you can with what you have.
There are better and worse answers, though, even when you don’t buy every assumption you sell. We have to give a number, but we shouldn’t give just any old number. We should do the best we can. If we’re going to put lipstick on the pig, we might as well choose a flattering shade.
Anyways… this got me thinking about whether there are similarly easy-to-use assumptions…
It turns out it is really easy to generate alternative assumptions in the same vein. Parallel trends is a stability assumption, and you can replace it with another stability assumption.
I thought up one I kind of like that formalizes analyses I (and many others) have done before. I find the argument generally more convincing/easier to convince myself of than parallel trends.
The setup:
Z = 1(t >= s) for some s, indicates treatment, i.e. the treatment launches at some point (it doesn’t have to hit everyone, but it’s fine if it does).
We observe Y(t) both before and after s.
Y(t) = Z Y(1, t) + (1 - Z) Y(0, t), where Y(1,t) is the outcome under treatment and Y(0,t) is the outcome without treatment.
We also observe another vector-valued series, X(t), with potential values X(0,t) and X(1,t).
Two assumptions:
E[Y(0, t) | X(0, t) = x] = m(x), i.e., the conditional mean function is stable over time without a structural break. (Stability)
X(0,t) = X(1,t). (Unaffected Predictors)
The first assumption is “testable” in the sense that parallel trends is “testable.” The scare quotes are because the identifying assumption—that the structural break does not occur at time s or the trends would have continued in parallel—is not testable, but we can at least test for instability in the series before the treatment and take evidence for its prior stability as evidence that it would have continued to be stable in the absence of treatment.
A nice practical thing about this structure is that doing this sort of test is very straightforward. There is a vast literature on structural break tests, so you avoid the frequently ad hoc parallel trends tests. Just run structural break tests (sup/ave/exp-F, qLL, or something) on the pre-period model to build the case for stability.
The bite of this identification strategy is how the first assumption combines with the second. The covariates with a stable conditional mean need to be unaffected by the treatment; i.e., we can’t just pick random “predictors” and say, “Hey, the prediction function is stable.” We need a predictor that isn’t affected by the treatment. A few scenarios where it’s highly credible:
If X(t) includes lags that predate the treatment, then they can’t be affected by the treatment.
More generally, X(t) can be a characteristic of the individual/group that is fixed pre-treatment.
X(t) can be a metric from a group that didn’t receive the treatment.
Given Stability and Unaffected Predictors, we can identify the average treatment effect on the treated ATT because:
m(x) can be identified from the pre-treatment data.
E[m(X(1,s))] = E[m(X(0,s))] = E[Y(0,s)] by the law of total expectation.
We can identify E[Y(1,s)] on the treated population from the data.
So, we have E[Y(1,s)] - E[Y(0,s)] = ATT.
The parallel trends assumption is:
Let G = 1 be the treated group and G = 0 be the untreated group.
In the absence of the treatment, the G = 1 group would have grown in parallel to the untreated group, i.e. E[Y(0) | G = 1, Z = 1] - E[Y(0) | G = 1, Z = 0] = E[Y(0) | G = 0, Z =1] - E[Y(0) | G = 1, Z = 0].
ATT = E[Y(1) | G = 1, Z = 1] - E[Y(0) | G = 1, Z = 1]. We don’t directly observe the second term, but we can fill it in using the assumption above.
ATT = E[Y(1) | G = 1, Z =1] - E[Y(0) | G = 1, Z = 0] - (E[Y(0) | G = 0, Z =1] - E[Y(0) | G = 1, Z = 0]), i.e. the familiar differences-in-difference.
The odd thing about this assumption, the more you squint at it, is that it requires the slopes to move at exactly the same scale. Yes, you can just adjust this assumption to assume they move proportionally and add covariates, etc., but then that’s another pretty strict assumption that doesn’t use any data on how the series typically move together—which is data you have!
That’s why I’m leaning towards something like the Stability assumption above to replace this. I want to use the available pre-treatment data to understand the relationship between treated and untreated groups, not impose the relationship.
There are two mistakes:
Not giving a number when the business needs a number.
Not caring what the number is.
“It doesn’t matter what analysis we do,” and “no analysis is good enough” are alternative forms of nihilism, and nihilism is lame.
There is nothing magical about a difference-in-differences. It’s just an assumption, like any other, and you can replace it with other stability assumptions that fit your situation better.
Don’t force parallel trends. It’s easy enough to pick something else if you make your identification argument from first principles. Not by just running a “method.” A vast array of stability assumptions can replace parallel trends.
You can think of an alternative that gets you a little closer to the truth, and that’s the whole game, isn’t it?
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn