In economics, is double differencing the best way to reliably assess the effect of an event when experimental methods are not available?

Double differencing is a useful quasi-experimental method for assessing the effect of an event, but it can lead to misleading results if the parallel trend assumption is not met. It is important to consider its limitations and caveats when applying it.

 

In economics, it is often necessary to evaluate the effects of events for evidence-based policy discussions. This process is crucial for solving social and economic problems and demonstrating the validity of policies. In particular, it is key to clarify whether the introduction of an economic policy or social program has actually had a positive effect, or whether it has had unintended side effects. Evaluating the impact of an event means comparing the outcome after the event to what would have happened in the absence of the event. This comparison provides essential information for policymakers, informing the design of future policies, and ultimately contributing to the well-being of society as a whole.
However, since hypothetical outcomes are unobservable, the effect of an event is evaluated by comparing the outcomes of the treatment group, which is composed of individuals who experienced the event, with the outcomes of the comparison group, which is composed of individuals who did not experience the event. The composition of the comparison and treatment groups is an important factor in determining the accuracy of the evaluation. If the two groups differ on factors other than the event, these differences can skew the evaluation results. The key to this is to create two groups that have no reason to differ in outcomes other than the event. For example, when evaluating the effect of an event on wages, you want to make sure that the average wages of the treatment and comparison groups would have been the same in the absence of the event. Ideally, an experimental method would design the event so that samples are randomly assigned to the two groups. However, this is often not possible when dealing with human samples or social issues.
Because of this difficulty, quasi-experimental methods are often used in situations where experimental methods are not available. Difference-in-Differences (DID) is a popular quasi-experimental technique. DID evaluates the effect of an event by subtracting the change in the comparison group from the change in the treatment group. It evaluates the effect of an event based on the parallel trend assumption that a change of the same magnitude as the change in the comparison group would have occurred in the treatment group even in the absence of the event. If this assumption is met, the two groups do not need to be formed so that their pre-event status is the same on average.
The usefulness of the double difference method has been recognized not only in economics, but also in various social science studies. As for its historical origins, it is said to have been first used by John Snow in 1854. He noticed that residents of the same neighborhood in London were receiving water from two different water companies. Only one of the two companies changed the water source, and the residents didn’t know which one. “By comparing the changes in cholera mortality rates before and after the switch between those who switched and those who did not, Snow concluded that cholera is transmitted through water, not air. This shows that double difference is more than just an economic analysis technique, and can be a powerful tool in other fields, such as public health. In economics, the method was first used in the 1910s to determine the effects of introducing a minimum wage.
However, when using the double difference method, it is important to ensure that the parallel trend assumption is met. If the parallel trends assumption is not met, applying the double difference method will misleadingly assess the effect of an event. For example, when evaluating the employment-growth effects of a worker training program, the parallel trend assumption would not be met if the treatment group had a larger share of workers in an industry with rapid job losses than the comparison group. This does not mean that setting the comparison group at a time before the event to increase the statistical similarity of the samples between groups will guarantee that the parallel trends assumption is met. For example, if the change is cyclically sensitive, such as employment, the simultaneity of the change may be more important in meeting this assumption than the statistical similarity of the samples between groups.
To make the application of double differencing more reliable, it is important for researchers to construct multiple comparison groups to ensure that the evaluation results are consistent across each of them. These methods can increase the confidence in an assessment that applies a double difference method. They also reduce the likelihood that the parallel trends assumption will be threatened by constructing comparison groups with high statistical similarity to the treatment group on a number of characteristics. These methods are particularly important in social science research, where experimental methods are difficult to apply.
Double difference is a powerful analytical tool that can be used in a variety of fields, including evaluating policy effects, assessing business strategies, and analyzing the effectiveness of educational programs. However, it is important to carefully examine the appropriateness of the parallel trends assumption before applying it, and to use it in conjunction with other complementary methods when necessary.