More on Google Prediction Markets

As some of you may know, Nick & I work for the wonderful, crazy company known as Google. One company policy is that employees can spend 20% of their time developing whatever new ideas they find interesting. I had a ton of ideas when I first showed up, but when I saw that a group had started working on an internal prediction market system, I tossed them and hopped on board.

They'd already done most of the coding, so it was awhile before I contributed. But after the system was live for a quarter and had gathered data, it needed to be analyzed. My 80% job is in Evaluation (measuring web search quality), so this was my department. Some of the results of that evaluation are in our Google Blog Post.

The trickiest problem was how to measure predictiveness. In our system, if an event has a price of ten cents, that means it should be 10% likely. This is because each outcome is worth 1 Gooble (play money dollar) if it comes true, but 0 otherwise. So if an outcome has a 10% chance of happening, its price should be ten cents.

But in the end, it will either happen or not - so how to evaluate the accuracy of that 10%? We came up with the idea of taking all the predictions (every average weekly price for each outcome) and bucketing them, ie 0-10%, 10%-20%, etc. Even though a single data point tells us nothing, if we collect 100 data points that we say should happen 10%-20% of the time, on average about 10-20 of them should happen.

The results are shown in the graph above. If events in each bucket happened exactly as often as predicted, the two lines would be the same, and you can see they are quite close. Personally, I think the level of accuracy is amazing, given that its a play money market, and these predictions were made as early as 16 weeks before expiration. Of course, its a play money market where people are trading about the things they work on, which helps. And this paper [PDF] suggests that play money markets can be accurate. But it is still quite nice to see that prices correspond to probabilities so accurately.

While this may seem like an odd project for a search engine company, the company's stated mission is "to organize the world's information and make it useful", and prediction markets fit right in. The information that we're organizing here is the knowledge and analysis embedded in the opinions of employees. And we make it useful by translating the fuzziness of belief into solid numbers. The idea is to aggregate knowledge from across the company into good predictions about various milestones and projects.

If you are interested in learning more about prediction markets, the most important thinker on the subject is Robin Hanson, see Could Gambling Save Science?, Shall We Vote on Values, But Bet on Beliefs? [PDF], and Idea Futures: Encouraging an Honest Consensus [PDF].

Share this

i>Does that include the

i>Does that include the janitors?

Where do you think that robot vacuum cleaner came from?

This Week's Carnival Of The

This Week's Carnival Of The Capitalists
It's a bit late to mention it, but this week's Carnival of the capitalists is up at AnyLetter as usual, here are my picks of the week:

Once the concept is proven

Once the concept is proven at the company level, does Phase II kick in and you introduce an external prediction market called "google markets (beta)"?

Congrats on your interesting

Congrats on your interesting market, and thanks for the plug. I hope you guys will consider creating conditional markets to more directly advise Google policy decisions.

"20% of there time" Does

"20% of there time" Does that include the janitors?

If we ruled the world...

If we ruled the world...
Google ritar upp sina planer f?r v?rldsherrav?lde p? n?gra whiteboards i ett av husen i Google HQ, Mountain View....

Brian: That Google policy

Brian: That Google policy applies to all the engineers. The robot cleaning device referred to in a joke comment can be found here. I've got one; it works quite well.

Sorry, not familiar with

Sorry, not familiar with what vacuum you are talking about ... and was it really the janitors that designed it, or does developing new ideas include "coming up with your replacement".

Hey there .... my MBA

Hey there .... my MBA classmates and I actually did a study along these lines for the 2004 presidential election. I would be happy to send along a copy of the report .... just shoot me an e-mail at kevinjdaniels@gmail.com

Well.. what do you know. I

Well.. what do you know. I authored a post on prediction markets - partly inspired by a short story by Robert Reed (The Opal Ball).
http://loxos.blogspot.com/2005/09/prediction-markets-real-life.html

Google Prediction

Google Prediction Markets
Patri Friedman, a google engineer who works on evaluating search quality, posted about the surprising accuracy of Google's Internal Prediction Market. I've written a post about the previous prediction markets workshop at SuperNova2005, which gave som...

Great to see Google getting

Great to see Google getting into prediction markets; certainly makes sense. Nice/interesting work. Results look great. Well done. Thanks for citing the "Does Money Matter?" paper. Would love to learn more about the software, the nature of the markets, the number of markets, etc.

One small comment:

From your post:
> "We came up with the idea of taking all the predictions ... and bucketing them"

This is a good/natural idea, and is actually a typical/common way to evaluate prediction markets; for example essentially the same procedure was used (not for the 1st time) in this 2001 paper:

http://artificialmarkets.com/am/pennock-2001-science/

Internal Prediction Market @

Internal Prediction Market @ Google
Google joins HP and the Pentagon in building an internal prediction market. this is very interesting ... putting Wisdom of Crowds into action...

Brokers, Bookies & Browsers:

Brokers, Bookies & Browsers: Yahoo’s R&D Insight
On the back end and the bottom line, gambling and investing converge, as both are tapped for smart-mob powered search …

...

flymix They?d already done

flymix
They?d already done most of the coding, so it was awhile before I contributed. But after the system was live for a quarter and had gathered data, it needed to be analyzed. - Rather interesting thoughts on this.