How Accurate Are Political Pundits' Forecasts? Research Says Not Very

Political pundits have often been accused of blowing hot air. Now there is some research to support that thesis. A group of Hamilton College students analyzed the accuracy of the predictions of a sample of pundits. Among the findings, those with law degrees are less accurate than others and liberals are slightly more accurate than conservatives (though their political success rate recently leads one to conclude that knowledge does not necessarily translate into effective action...). Hat tip Chris Rowe

We report on the first-ever test of the accuracy of figures who made political predictions. We sampled the predictions of 26 individuals who wrote columns in major newspapers and/or appeared on the three major Sunday television news shows (Face the Nation, Meet the Press, and This Week) over a 16 month period from September 2007 to December 2008. Collectively, we called these pundits and politicians “prognosticators.” We evaluated each of the 472 predictions we recorded, testing it for its accuracy. Based on an analysis of these predictions, we answer three questions:

1.Which prognosticators are most accurate? We found wide disparities in the predictive accuracy of these individuals, and we divided them into “the good, the bad, and the ugly.”

2.Which characteristics are associated with predictive accuracy? We examined the effects of age, education, ideology, and other factors on accuracy.

3.What is the purpose of media pundits? We discuss whether the ordinary citizen should look to pundits for deeper analysis of events, or whether pundits are simply a more enjoyable way to learn about the events of the day. We also consider alternative viewpoints, including the notion that pundits are useful as representatives of opposing points of view in the country, and the idea that they are simply entertainers.

We discovered that a few factors impacted a prediction's accuracy. The first is whether or not the prediction is a conditional; conditional predictions were more likely to not come true. The second was partisanship; liberals were more likely than conservatives to predict correctly. The final significant factor in a prediction's outcome was having a law degree; lawyers predicted incorrectly more often. (R-square of .157) Partisanship had an impact on predictions even when removing political predictions about the Presidential, Vice Presidential, House, and Senate elections.

A number of factors impacted whether a prediction was extreme. In other words, we measured whether certain things led to pundits saying that an event absolutely will or will not happen. Again, conditional predictions were significant, and made something more extreme. Predictions about the GOP primaries were less extreme, as were predictions about the Vice Presidency. Predictions about the partisan makeup of the House were more likely to be extreme. As prognosticators aged, they became less extreme. (R-square of .145)



We have discovered a number of implications from our regressions and analysis of the data. First, we have discovered that six of the analyzed prognosticators are better than a coin flip (with statistical significance.) Four are worse, and the other 16 are not statistically significant. A larger sample can provide better evidence addressing the question of if prognosticators on the whole are better than a coin flip. We understand that being better than a coin flip is not a high bar to set, but it is a serious indictment of prognosticators if they are, on average, no better than a flipped coin.

According to our regression analysis, liberals are better predictors than conservatives—even when taking out the Presidential and Congressional election questions. Whether this holds true from election season to election season needs further evaluation; liberals may have implicit benefits from Obama winning the 2008 election. Tentatively, however, we can assert that liberals are better predictors than conservatives. Additionally, individuals with law degrees were less accurate than those who did not possess law degrees.

A final important implications is that we did not discover that certain types of predictions tended to be more or less accurate than others. For example, we did not see that economic predictions were more accurate than healthcare predictions. This suggests that prognosticators on the whole have no unique expertise in any area—even on political predictions, like the Presidential or party primary elections.