Inside the Paradox of Forecasting:Consumer Beware

Humans have an inherent desire to reduce risk, in part by attempting to foresee the future. Gurus, forecasters, media mavens - bloggers - all attempt to address this need by providing insights. Some are based on data, some on experience, some, well, who knows.

In his Big Picture blog, Barry Ritholtz today highlights a story on Nouriel Roubini, the guy who 'called' the Great Recession of 2008. What the story underscores is that however intangible, ineffable, evanescent or immaterial, there is usual a reason why someone gets things right - once - and may not do so again. All managers, particularly marketers, should constantly be searching for the reason behind an expert opinion.

From The Big Picture:

In yesterday’s afternoon reads, I mentioned a Boston Globe story “That guy who called the big one? Don’t listen to him.”

While it ostensibly looks at the track record of Nouriel Roubini, it isn’t really about him –its really about outliers and mean reversion in forecasting.

“How can someone with the insight to be so right about a major event be so wrong about so many other ones? According to a recent study, it’s simple: The people who successfully predict extreme events, and are duly garlanded with accolades, big book sales, and lucrative speaking engagements, don’t do so because their judgment is so sharp. They do it because it’s so bad . . .

[Oxford economist Jerker Denrell and Christina Fang of New York University] are the latest in a long line of researchers dismantling the notion that predictions are really worth anything. The most notable work in the field is “Expert Political Judgment” by Philip Tetlock of the University of Pennsylvania. Tetlock analyzed more than 80,000 political predictions ventured by supposed experts over two decades to see how well they fared as a group.

The answer: badly. The experts did about as well as chance. And the more in-demand the expert, the bolder, and thus the less accurate, the predictions. Research by a handful of others, Denrell included, suggests the same goes for economic forecasters. An accurate prediction — of an extreme event or even a series of nonextreme ones — can beget overconfidence, which can lead to making bolder and bolder bets, and thus, more and more errors . . .

There’s no great, complex explanation for why people who get one big thing right get most everything else wrong, argues Denrell. It’s simple: Those who correctly predict extreme events tend to have a greater tendency to make extreme predictions; and those who make extreme predictions tend to spend most of the time being wrong — on account of most of their predictions being, well, pretty extreme. There are few occurrences so out of the ordinary that someone, somewhere won’t have seen them coming, even if that person has seldom been right about anything else.”