More and less theory, please
Chris Blattman links to an upcoming JEP article on the need for more theory in field research (article here).
As a general call for more theory, I am skeptical. Let us consider two phases of experimental research: design and analysis. In the design phase, I agree that more theory can help motivate more nuanced experimental designs. I agree with the authors that writing down theories can help us see ways to play competing theories against each other or add valuable manipulations to the design. If it helps you to write down structural equations during the design phase, all the better. Prior to randomization, we can motivate our designs as we see best.
I part ways with the authors, however, when it comes to using theory in the analysis. Specifically, using theory in the analysis phase is a form of mediation analysis — attributing causal interpretations in an experiment where randomization does not justify them. Let us say we have three quantities we can measure A, B, and C. Theory tells us that if A is on, then B turns on, and then C does so too. If A is off, then B is off, and then C is off. In the experiment we were able to manipulate A. We observe that B and C behave as we expected. What then can we say about our theory and the relationship between B and C?
Randomization of A gives us a strong position to argue that turning on A leads to turning on B. It also gives us a strong case for A turning on C. But the design (randomizing A) says nothing about the relationship of B and C. It may be that A is linked to some unobserved variable D that influences both B and C, but they have no direct link to each other. It may be that C causes B, but not the other way around. In this scenario, theory could have been helpful in the design phase. For example, manipulating both A and B would tell us something about the relationship of B with C. But in the analysis phase, talking about the relationship between B and C is no longer causal. We are back in the realm of observational work.
This is not meant to be derogatory of observational work, but simply let us be clear about when we have causal interpretations of relationships and when we do not. Basing the analysis of experiments on theory jeopardizes the causal claims we wish to make.
