The predictive market at hubdub.com is now a few weeks old and I've participated in several settled questions. Basically the way the market works is someone posts a question and sets some initial odds for the answers. People can then bet pretend money on the outcome and the payoff is based on the odds. The odds are adjusted based on the betting. The questions are settled by administrators after the outcome or answer is published. You can cash in your bets before the settlement and the cash in value is determined by the odds at the time of cash in. So it is possible to place a bet and then cash it in for more before the bet is settled thus making a profit. Based on how much money some people have made in the short amount of time it is open, it appears that there is a lot of active trading. I don't really intend to do any active trading, I'm more interested in how often the crowd is correct and am making that my experiment. According to the book The Wisdom of Crowds by James Surowiecki, I'm not sure if the predictive market in this setup will work or not [1]. One of the things Surowiecki identifies in the book is the cascade effect. In order for predictive markets to work people must be acting independently, where each person brings a little bit of their own knowledge to the collective wisdom. In the cascade effect, people know how other people feel, and their own opinion is influenced by this [1].
Since this is a market where the bets change with time as more information is revealed, the end resulting percentages almost always match the outcome. For example there was a question on who would be the MVP of the Superbowl this year. Before the game, Tom Brady was quite a bit ahead. As the game progressed, Wes Welker of the Patriots surged into the lead briefly, but then as the game ended Eli Manning was in the lead. So even though the end result matched what happened (Eli was named the MVP), it was pretty obvious toward the end of the game that he would be so named, and so people changed their bets accordingly. So as I tally results for whether the crowd made the right prediction, I will always assess what the crowd thought before an event starts. In this example, the crowd predicted that Tom Brady would be the MVP and was wrong. Here are the outcomes I wil keep a tally of:
The crowd is correct: More than 60% of the crowd predicts the correct outcome
The crowd is wrong: Less than 40% of the crowd predicts the correct outcome
It's a draw: Between 40% and 60% of the crowd predicts the correct outcome
Here are my tallies so far:
So far the crowd's won-loss-draw record is 1-3-1.
[1] Surowiecki, James. 2004. The Wisdom of Crowds. Anchor Books: New York.
Doesn't look like a good start for our forecasting ability! I expect over
time that will improve. One of the best ways to assess accuracy is to
compare the % outcome of events with similar forecast probabilities. Google
did a very nice analysis of the accuracy of their internal prediction
markets. Their analysis is here:
http://googleblog.blogspot.com/2005/09/putting-crowd-wisdom-to-work.html