Ten Commandments Posted in Griffin Courthouse, Spaulding County, Georgia
shadows from right to left--Mark Lyttle, his wife, and me
An opinion piece is a great way to stir up a conversation and a lousy way to present arguments in scholarly detail or provide sources. Here are a few other thoughts related to "Political Scientists are Lousy Forecasters," published in hard copy of the New York Times Sunday Review on June 24, 2012. (I am not under the impression that a whole lot of people are interested, but for folks with questions, it seemed more efficient to do this here than on email.)
Auto- Q and A
Q. Why did you write this?
A. Because a couple weeks ago I returned from a day in Judge Fletcher Sam's court in Griffin, Georgia and listened to an impassioned speech just outside his court room by Sernita Trice, a one-time political science major at Penn State, that began, "There is no justice here. No one cares about us." The details are not relevant here, but she was right and I was embarrassed for my discipline for ignoring a myriad of politically and intellectually gripping questions that are in cities like Griffin and counties like Spaulding and states like Georgia throughout our country.
I returned to my room at the Griffin Inn and that's when I saw e-mails from my chair and APSA urging me to defend political science against being cut from the National Science Foundation budget. Of course, I've had my frustrations about political science research agendas long, long before this but it was the specific conjuncture of frustration and the timing of the appeals for NSF support that moved me to write this.
Q. Geez! I thought we'd gone beyond that old Wolin v. the behavioralist debate. Why are you stirring up that old can of worms?
A. I'm not. I disagree quite strongly with both the claim of quantitative political scientists to be emulating the method of Karl Popper and also Sheldon Wolin's attack on them for doing so. As I point out in the piece, Popper rejected behavioralism; he also rejects probabilistic research's claim to being scientific knowledge. I've published chunks of this in the Methods Appendix to States Without Nations: Citizenship for Mortals (Columbia University Press, 2009).
Also, I've criticized political theorists for making non-falsifiable claims about the alleged decline of the nation-state, including Wendy Brown's Walled States, Waning Sovereignty (Verso, 2010).
Q. Does this mean that there's a rift between the theorists and everyone else in your department?
A. NO! My colleagues are intellectually curious, well-read, and great interlocutors, as well as supportive of my less traditional research agendas. It's true that I've been trying to encourage more reflection on the implications of Tetlock's research as well as more general questions about our scholarly objectives and I see this piece as part of our ongoing conversation.
Q. You really go after James Fearon and David Laitin. What's that about?
A. The quantitative civil wars studies industry reveals various troubling and even horrifying commitments, especially to reductionism, inductivism, and the axiomatic status of the nation and ethnic groups. These studies radically impair our society's ability to grapple with pressing problems generated by kinship groups in this country and elsewhere.
Unfortunately there wasn't room to go into the many problems with Fearon's and Laitin's work. Here's a link to a work in progress:"A Popperian Reading of Civil War Studies," is what I think I'll call it. It received comments from a peer reviewed journal with a recommendation that I revise and resubmit but this all happened during the time-frame when I was moving and starting new projects so I haven't yet followed up.
Q. Also, that whole Tetlock metaphor with the chimps throwing darts: isn't that a bit unfair? It's such a provocative image.
A. Yes, it is a provocative image. Thanks, Professor Tetlock, for providing this! All representations of ideas from math and indeed all observations use metaphors. Words, ideas, and symbols that are vivid, accessible, and clarifying are better than others.
Q. What are the sources you're using for the claims in "Political Scientists Are Lousy Forecasters"?
A. Here you go: a list based on the materials I put together for the intrepid New York Times fact-checker.
Q. Any reading recommendations?
A. Yes. There are lots of greats methods work. Alas, I don't have the time for putting together a full list but I do want to plug this one: Kristin Monroe put together an amazing collection of essays that should be required reading for every political scientist, and is available on Amazon for $12!: Contemporary Empirical Political Theory (University of California Press, 1997).
Q. Did you have to make any cuts?
A. Alas, several. The ones I feel the worst about are the places describing examples of terrific work my colleagues are doing. Here they are:
Intriguingly, another article cautions against taking at face value what people tell survey researchers, and not just on hot button issues such as racial attitudes. In an ingenious study, funded by the NSF Digital Government Research Program, not part of the Political Science division, political scientist Michael Neblo and his colleagues found that people in their sample who were angry about politics were substantially more likely to express a disinterest in meeting their members of Congress when asked about this as a survey question than when researchers arranged actual online meetings with their representatives.
.... Government can and should assist political scientists, especially those who use history and theory to yield insights beyond the daily headlines. [THE LAY READER DOESN’T NEED THIS such as Jeffrey Winters’ analysis of oligarchy in the United States, Charli Carpenter’s reflections on the "everyday politics," shaping political science research, Dorian Warren’s work on the intersection of race and labor politics, and other articles appearing in the APSA’s Perspectives on Politics, a journal offering readable, relevant expert analyses that orients us to shifting political contexts and challenges our intuitions.CUT]Q. I was reading other stuff on your blog. Why do you have all these posts on U.S. citizens being detained and deported?
A. This is a little experiment in be-the-change, politically and intellectually. I'm drawing on these very specific people and their stories to engage theories of citizenship and membership and also to throw a few monkey wrenches into the system producing these outcomes. (Hmmm, wonder how political scientists compare with monkeys throwing their wrenches.) I focus on the plight of U.S. citizens being unlawfully detained and deported because they're the 900 pound gorilla in the mine: if even the rights of U.S. citizens are not being protected then that tells us a lot about the treatment of everyone else.
Also, anything the government does to protect due process rights for U.S. citizens will improve the rights of everyone else -- because the only way to establish citizenship is to provide people with assigned attorneys. (People may not realize they are U.S. citizens or be in a position to file the right motions to stop the deportation machine, especially if they are 19 year-old guys just being released from jail for stealing a car and they are whisked straight from the jail lobby to an ICE detention center without anyone telling their families.) If you're interested in more about this, please see "U.S. Government Unlawfully Detaining and Deporting U.S. Citizens as Aliens," Virginia Journal of Social Policy and Law, 18:3 (2011), 115 p.
In addition, I'm supervising a Deportation Research Clinic. The major project is to conduct a survey of misconduct in deportation proceedings. If any of this is of interest and you want to collaborate, please be in touch.
Q. Anything else you are working on?
A. My major work-in-progress is a long-term project, "200 Per Cent American." This is the response Mark Lyttle gave to a deportation officer who asked if he was a U.S. citizen after he was detained in the Atlanta airport on his return from Guatemala City. Mark, who is cognitively disabled and bipolar, had been deported to Mexico and then from there deported to Honduras. It took him more than four months before a consular officer figured out his story and issued him a U.S. passport--and it was this passport that the government said was fraudulently obtained when he returned and had the interview. (Mark's attorney was unable to obtain his release from ICE custody and emailed me. I was at a spring workshop in Dartmouth and called an ICE agent in Washington D.C. with whom I was working on another story, and she called the Atlanta ICE office and procured Mark's release.)
The book I'm writing is a story about his experiences told through the historical and narrative lens of Miguel Cervantes, Don Quixote, a novel making fun of the Spanish conquistadors for acting on the basis of the crazy myths that led them to the Americas.
Q. Where are you?
A. I'm dividing my time between Chicago and New York City this summer.
UPDATE (June 26, 2012)
Q. What has been the response to your piece?
A. A deluge of grateful emails, including from political science NSF panel advisers -- the plurality of which literally say, "the emperor has no clothes"; and ferocious, incoherent, illogical, and often ad hominem blogosphere attacks from the naked emperor's fashion consultants desperately trying to convince the elite guard that the finery really is there.
(Clearly an email that says "the emperor has no clothes" won't work; these statements only work when the spectators/chorus do this together and thus embarrass the emperor's henchmen into abandoning their latest round of propaganda and group-think activities.)
I will let the dust settle a bit and then respond sometime next week in a separate post.
10 comments:
Dr. Stevens: Your NYT article on PoliSci forecasting was very interesting. Especially the lack of forecasting of the end of the Cold War, and of the rise of the Middle East since 9/11, has been apparent to me for many years. I am glad someone from within your field finally noticed it. Sorry to appear self-righteous. I come at one point from the academic world, though not now, and am very conscious of its tendency to prognosticate but not to repent. It is a failing. Geoffrey White, Iowa
Professor Stevens, I would be interested to hear your response to Henry Farrell's blog post over at MonkeyCage. Especially pertaining to his point that you misrepresented the Tetlock work as a study of political experts "most of whom were political science PhDs"--when in fact Tetlock just says a majority of the experts had doctorates in disciplines like political science, among others (economics, law, business, public policy, journalism).
To respond to the previous comment:
1) I read the passage on the expert backgrounds too quickly. Tetlock writes: "Participants were highly educated (the majority had doctorates) and almost all had postgraduate training in fields such as political science (in particular, international relations and various branches of area studies), economics, international law and diplomacy, business administration, public policy and journalism."
NOTE: Professor Tetlock was kind enough to share with me more specific characteristics of his sample. According to an email from him, between 65% to 80% of the PhDs in his sample were in political science, or 34% to 42% of all those surveyed, and not "most."
I regret this mischaracterization of Tetlock's sample.
2) Tetlock in his correspondence and elsewhere in the book makes it clear this is irrelevant to my argument: political scientists were indistinguishable from the rest of his sample: "It made virtually no difference whether participants had doctorates, whether they were political scientists, journalists, or historians, whether they had policy experience or access to classified information, or whether they had logged many or few years of experience in their line of work. As noted in Chapter 2, the only consistent predictor was, ironically, fame, as indexed by a Google count: better-known forecasters--those more likely to be feted by the media--were less well calibrated than their lower-profile colleagues." (Fearon and Laitin would be super high on this list, by the way.)
"NOTE: Professor Tetlock was kind enough to share with me more specific characteristics of his sample. According to an email from him, between 65% to 80% of the PhDs in his sample were in political science, or 34% to 42% of all those surveyed, and not "most."
Q: What % of the polisci PhDs surveyed were academic political scientists? Out of these, what % training in quant. methods? Out of these, what % had expertise modeling and predicting events/behavior relevant to the questions posed by Tetlock?
A quote comes to mind:
Edward C. Banfield: "Social scientists should never try to predict the future: they have hard enough time predicting the past." http://edwardcbanfield.wordpress.com/
Cheers!
Kevin R. Kosar
I'm glad see the question of the relationship between (a) funding for political science research and (b) the issues being researched and methods used appear so prominently in the NYTimes op-ed page. Congratulations! I too "look forward to seeing what happens to my discipline and politics more generally once we stop mistaking probability studies and statistical significance for knowledge."
I would recommend those interested in this issue to also check out Theodore Lowi's important article, "The State in Political Science: How We Become What We Study" The American Political Science Review , Vol. 86, No. 1 (Mar., 1992).
I'm glad see the question of the relationship between (a) funding for political science research and (b) the issues being researched and methods used appear so prominently in the NYTimes op-ed page. Congratulations! I too "look forward to seeing what happens to my discipline and politics more generally once we stop mistaking probability studies and statistical significance for knowledge."
I would recommend those interested in this issue to also check out Theodore Lowi's important article, "The State in Political Science: How We Become What We Study" The American Political Science Review , Vol. 86, No. 1 (Mar., 1992).
Dr. Stevens,
Thank you for your interesting op-ed and starting an important conversation on the future direction of political science.
I am quite interested by your paper that attempts a Popperian interpretation of the civil war literature. It is quite innovative. I did, however, have a question that I wondered if you could answer.
Popper traditionally ascribed the social sciences to the realm of "pseudo science"; that is, they did not have the characteristics to be considered a science. Under Popper's own schemata, one shouldn't use falsification as a criteria to evaluate pseudo science, only sciences.
Furthermore, I don't think any proponents of using quantitative models in political science claim that it has the stature of a science either. Thus, why are you applying falsification, a standard for science, to a pseudo science (social sciences)in the paper? Thanks.
By your definition of science, clinical medicine isn't a science. It can only predict probabilistically. Chemotherapy kills some people. On average we hope it cures more people than it harms. But for some cancers, it is a very close run thing.
More generally, if social scientists want to do science, they need to run more experiments. The get-out-the-vote literature in political science is as rigorous as any work in clinical medicine. The problem is the focus on "big" questions that scholars like yourself push. They cannot be answered scientifically because of the inability to perform experiments. But if social scientists focus on answerable questions, people like you will complain. But in science, there are no awards for asking questions you cannot answer.
Hey Jackie, your article in Sunday's Times was a pleasant surprise for me. I wonder if you happened to read Nate Silver's response ( http://nyti.ms/LzR0vC ). I really like Nate Silver's analysis generally, and I would really love to hear your response to his piece especially since it seems like he may have missed your point. As you said above, an op-ed is a very constrained but I would have liked to have seen you anticipate Silver's criticisms and head them off at the pass because I feel like you and he would agree on a lot.
Anyway, hope you're doing well. I'm glad to read you're still working on the "200% American" project.
Nate
Post a Comment