Probabilities, Prediction Markets, and Popular Fallacies
With Hillary's surprise victory over Obama in the New Hampshire primary, pundits everywhere are decrying the allegedly 'wrong' odds that prediction markets like Intrade were displaying prior to the announced results. (As just one example, Barry Ritholtz weighs in with his 'explanation' of : "Why Opinion Markets Fail" )
At one point the betting markets were implying over a 90% probability for Obama to win. Does this mean they were 'wrong'? No it does not. It is impossible to judge whether a given probability is/was correct based on the outcome of a single event.
A 90% probability simply implies that, if you encounter a series of events each with a 90% probability, then 9 times out of 10, the favored outcome will occur; and 1 time out of 10, the unfavored outcome will occur. Those like Ritholtz who are now calling the prediction markets 'wrong' are implying the following: if the probability is 90% for an outcome to occur, then that outcome should occur every time. In other words, if the odds are 90% in favor of something -- it should happen 100% of the time! But this is obviously fallacious. If the outcome occurs 100% of the time, then the correct probability to assign to it would be 100% -- not 90%.
To validly assess the accuracy of prediction markets, one needs to aggregate all the situations where the odds were 90%, and then calculate whether the favored outcome indeed occurred 90% of the time. (And do the same with each level of probability.) This -- and only this -- will tell you how accurate prediction markets tend to be.
posted by Rob Tarr at 4:52 PM
4 Comments:
As every good prognosticator knows, if you couch your forecasts in probabilities, the innumeric will never know you were wrong. Its a cheap trick for the easily fooled.
Imagine if instead of a "THE END IS NEAR" sign, every loon carried a sign that proclaimed
THERE IS A 57% CHANCE THAT THE END IS NEAR!!!
The fact that this didn't happen -- and the 43% probability did -- doesn't mean this forecast was accurate. It merely meant that the person had proferred two possibilities and one of those two occurred. But the math remains unverified.
Neat trick: By your definition, PREDICTION MARKETS CAN NEVER BE WRONG, so long as they maintain a 1% possibility of the alternative outcome.
That's hardly a satisfying defense . . .
Au contraire, I specifically said prediction market can be proven wrong, but not by reference to a single event.
For example, the probability of rolling a 5 on a six-sided die is 1 out of 6, or a 16.667% probability. That's a low probability. However, eventually I will throw a 5. When that happens, does that mean the 16.67% probability was "wrong" -- because the "unexpected" happened? Of course not. We know for a fact that the probability is exactly 1 out of 6.
The only way to prove that the probability is wrong -- i.e. *not* 1 out of 6 (e.g. if we suspect the die is loaded) -- is to throw the die hundreds of times and confirm the whether the frequency of a '5' does indeed occur 1 out of 6 times.
Same with prediction markets. If you want to prove that they are 'wrong' or 'inefficient', you need to look at the frequency of results over time, relative to their predicted probabilities.
I have no dog in this fight, such a study could very well prove that prediction markets generally assign incorrect probabilities. But you can't tell that from a single event.
By that definition, we can never "prove" the Democratic New Hampshire results were wrong, as we cannot rerun them 10X.
That is a deeply unsatisfying solution.
What I think this whole episode -- along with the many others I enumerate -- is to recognize that there are specific conditions where these markets don't perform as advertised.
IMHO, the burden of proof remains on the supporters of prediction markets to explain the anomalies, outliers and failures. Percentages fail to do so . . .
If we're talking probability: in an unknown distribution, a Bayesian approach is recommended. A failure on the level of NH strikes a blow to whatever prior you had.
[Followed through from BH blog--pretty sure he gets statistics.]
Post a Comment
<< Home