Kevin Clarke visits UIUC
The methodology seminar series kicked off the semester this week with a visit from Kevin Clarke of the University of Rochester. Clarke, with coauthor David Primo, is finishing on a book on the use of models in political science research.
The basic premise of Clarke and Primo is that models should be treated like maps. Like maps, models are not inherently true or false, but rather are useful for some purpose. Likewise, theories, as collections of models, are also outside the true-false dichotomy.
On the whole, I would say the idea was well received by the political science department at Illinois. Much of the debate concerned how to define “usefulness.” In the case of a map, there is usually a natural, physical use that provides a clear cut metric of usefulness. In political science, justifying the criteria upon which models should be judged is a more difficult task. A related criticism that was raised in the debate is that substituting “useful” for “true” is simply a semantic exchange. For the purposes of theory testing, a main focus of modern political science research, “useful” would be defined as the ability of the model to provide a clear test of an observable deduction, which we might in other discussions simply call falsification of the model.
Clarke’s response, and a theme of the book, is that actually testing a deduction of a theory requires a model itself. Political scientists are certainly familiar with the phrase “data do not speak for themselves,” but still consider a model to be a test based purely on the data. Clarke argues that the test itself is therefore based on a model of the data (though there was some debate whether pure experiments did allow the data to directly test a theory — I ultimately disagreed, even simple experiments require a model, but that is the topic for another post). In many ways, I found this part of his argument to be the most convincing: how do we test one model (scientific) with another model (statistical)?
The book is an ongoing work and is notably missing a final concluding section. Clarke indicated that this chapter would consider counter-arguments to the models as maps position. I look forward to reading his responses, as well as seeing more concrete applications of models in maps in political science. Clarke himself pointed out that change in the discipline will be hard (perhaps accomplished through replacement rather than persuasion), so I do not expect to see a his argument cited in all the articles in the next APSR.
Nevertheless, I do hope to see more research think carefully about the usefulness the employed models for purposes other than theory testing. In fact, selecting the best model for the research is the core of the argument in a working paper of mine, with John Ostrowski. In this paper, we argue that simulation studies of knowledge (in which researchers construct a counterfactual world of fully informed citizens) should employ models that are the most accurate at predicting responses in observed data. Previous studies used only linear models, while we consider a variety of linear and non-linear machine learning techniques. While the best linear model is as accurate as the best non-linear model, the non-linear models predict relatively little change under a fully informed population, indicating that political knowledge may not be as important to attitude formation as we previously thought.
In the immediate future, I see simulations (such as our paper or agent based approaches) as the most likely standard bearers for the Clarke and Primo position. Whether the models as maps position takes hold in a larger audience remains to be seen.
