Liveblogging: Philip Tetlock

Book CoverAbout to see a talk about Philip Tetlock, author of Expert Political Judgement: How Good Is It? How Can We Know?, and will report. The talk is entitled "Foxes, Hedgehogs, and Dart-Throwing Monkeys", so apparently this is going to be about some kind of brutal zoo death match. My money is on the monkeys for sure. Hedgehogs don't got a chance.

He's starting by talking about the political blogosphere - maybe this isn't about zoos. He is asking us the question: "What factors do you think most predict the success of a blogosphere pundit? The degree to which the pundit is accurate? Or passionately partisan and entertainingly sarcastic?". The audience picked (b) - no surprise. Back in the 19th century, someone like John Stuart Mill might have suggested that the marketplace of ideas will eventually self-correct. Tetlock's talk will be about why we should keep score of who has an accurate view of the world.

He's been surveying experts for a couple decades, asking them to predict with enough specifity and clarity that we can score them later. He says that there is an inverse relationship between what makes people successful in the blogosphere and what makes people accurate. He basically says we should grade talking heads, so they don't talk such bullshit. This guy is fucking awesome. He could whoop a monkey any day, even if the monkey had darts.

Besides scoring experts on accuracy, he gauges their willingness to change their minds when new information comes in. (A classic irrationality is to fail to update). All his participants made their living writing and thinking about political/economic trends, so journalists, NGO employees, intelligence analysts, think-tank employees, etc. Questions/outcomes need to be exclusive, exhaustive, and clear. Then you get people to place numerical probabilities on each set of outcomes for a question. Subjects included economic performance, government debt, defensive spending, wars, entry/exit from international agreements, leadership changes... He had about 30K judgements of about 3 outcomes each, so 90K outcomes.

Ha - question from the audience - "Are your results ever broken down by individual?" Tetlock replies - No way! People only cooperated because of anonymity. This is not about identifying winners and losers [Patri: It's about proving that everyone is a loser!].

He used a quadratic scoring rule, so it's incentive-compatible. He briefly mentioned the Kahnemann/Gladwell axis, about whether thinking or blinking works better, and said that he is much more on the Kahnemann side, that what ails human judgements is too much intuition, not too little.

This guys work must be right, because he says that I resemble the best forecasters in his sample, because I do well on the CRT, and classify myself as a fox who is wary of master theories. Unfortunately, while the best could beat random guessing, they could rarely beat simple extrapolation algorithms. People who believe in single principles that generalize to explain the universe (hedgehogs) do worse than most forecasters, but are over-represented among grand-slams. When something really wacky happens, a hedgehog usually predicted it. Of course, there are lots of hedgehogs, and their predictions are all over the map, so just by chance, this happens.

Relating this to the blogosphere, you don't make your reputation by making accurate calculations about what will probably happen, but by making wild predictions that occasionally turn out right.

Have I mentioned, yet, that this guy totally rules?

Risk factors for predicting poorly: Too quick to make up mind (low scores on CRT), too slow to change mind (hedgehog), too passionate about a particular political idea.

Question - "What about aggregating experts?". It turns out that if you aggregate hedgehogs, their biases average out, and they do about as well as aggregated foxes. The accuracy is not that much higher than the good individuals though, unlike in Wisdom of Crowds results.

Question - "What else matters? Liberal/conservative? Age? Academica/Gov" - Answer is, almost nothing. The CRT and the fox/hedgehog were the only strong predictors.

My Question "Prediction Markets" - He said "Yeah, they are great, they help people think more accurately. But how would they serve the public good of helping incentivize people to be more accurate?" My answer - Rafe Furst's Truth Markets. Basically, you make prediction markets in predictions that talking heads make. Furthermore, you have bond funds that track the performance of specific talking heads based on their public statements. And the performance of these bond funds indicates the accuracy of the wonk.

Comment: "But in the financial sector, results are tracked, and people invest in actively-managed funds anyway." (Yeah, but less than they used to)

I hung out afterwards and talked to him a bit about metrics for forecast performance and prediction markets. Good stuff.

Share this