Using Difference-in-Differences to identify causal effects of COVID-19 policies

Andrew Goodman Bacon, Jan Marcus

This paper reviews the challenges of performing Difference in differences analysis for studying the impact of COVID 19 pandemic policies in Causal inference.

Challenges

Packaged policies: multiple, similar policies are enacted around the same time and it is difficult to attribute the effect to one policy.
Reverse causality: the policy can be a reaction of the disease prevalence and stress to the health systems.
Voluntary precautions: people take precautions before policy gets enacted.
Anticipation: behaviors may change in response to information about the policy itself. (e.g., stockpiling & shopping before lockdown)
Spillovers: disease prevalence is correlated geographically and spill-over to nearby regions. Policies that help treatment group may also help the control group.
Variation in policy timing
Measurement and scaling of the dependent variable

Recommendations

So the recommendations are: 1. Estimate dynamics 2. Choose the control group wisely 3. be careful with regression DD 4. Sign the bias 5. Be clear about what is knowable