Selection Bias and Risky Strategies

Back when I was new to poker, I had a lot of ideas about metaphors from poker to strategy, economics, and life. But now that I mostly play on the internet, beating the games I know how to beat, I'm a lot less creative, it's more of a grind than an adventure. Still, this article from Mahalonobis on selection bias reminds me of the poker tournament world. The key is these two graphs:

Sample Selection Bias 1 - All Companies Sample Selection Bias 2 - Surviving Companies

You see that if you look at the performance of many businesses w.r.t. a risky practice that is a bad gamble, you can find the slightly negative trend line. But what happens if you consider only those businesses still around? This happens accidentally all the time - after all, its much easier to survey those businesses. The result is that you eliminate the worst failures of the practice you are examining, leaving a falsely positive impression.

The same thing happens in the poker tournament world. Certain styles of play trade EV for variance, allowing people to build up huge stacks occasionally, but usually go bust. Such players often win tournaments - but that doesn't mean they are playing right. How many times do they fail for each victory? Do they fail more often compared to the money they win than a more conservative player? Some of these "maniacs" are smart players, carefully choosing their gambles and maximizing their returns. But some of them, frankly, are just maniacs, gambling and getting lucky, and giving the false impression that high-variance play is the way to go, because we don't notice the hundreds of people playing that way and losing.

Not only does this cause people to pick the wrong role models, but it gives them a false idea of their own abilities. That one maniac who got lucky almost always thinks he's the new Phil Hellmuth or the reincarnation of Stu Ungar (but without the fatal drug habit). This often leads them to make poor financial decisions regarding their future "career" at poker - which is nice for the good players, but not so nice when their winnings trickle away.

Nor is this limited to poker or business. Consider an activity that usually goes fine, but has a small chance of resulting in a mysterious death. (Perhaps some backpacking or rock climbing practice). How are we ever going to hear about them? The only people who describe the practice are those who survived. In general, anytime people will tell you about success but not failure (or vice-versa), you are going to get a distorted view. This is why double-blind placebo controlled studies, or carefully tracking *all* data, are so important for getting at the truth.

While the biased apparent results of risky and poor gabmles is a common, direct consequence of survivorship bias, there are surely less obvious ones as well. If anyone has thoughts on similar classes of strategy that will get consistently distorted in report, I'd love to hear them.

Share this

It's the infamous "Dolphin

It's the infamous "Dolphin Effect".

According to legend, sailors who abandon a sinking ship will be pushed towards shore by our loving friends, the dolphins.

It is important to note that many fewer reports were received from those sailors who got pushed *away* from shore.

The perceived success of the

The perceived success of the U.S. is not a single observation, but a fairly long series. Rather than selection bias, naysayers should focus on confounding factors (e.g., Japan and continental Europe seemed to be outperforming the U.S. in the 1970's and 1980's, but that now appears to have been mainly due to favorable demographics; their more recent underperformance could be blamed on demographics rather than economic policy).

The idea that the strategies of the smaller Asian nations are variance-enhancing, rather than value-enhancing, is a good one. Neal Stephenson's fictional "Kinakuta" in Cryptonomicon is a sort of case study of this.

I meant to say "...that have

I meant to say "...that have tried to copy us have shown..." in the above, but I guess my thoughts ran a little too far ahead of my fingers which skipped a few words to catch up. IRTE.

Very true -- I wish more

Very true -- I wish more people were educated about basic statistics like this.

You see selection bias all the time in the financial world. Any index of hedge funds, stocks, or anything else is skewed to the positive side of the distribution for that reason. Do stocks really return 8-10 percent year after year? It depends on when you invested and which companies you chose.

It is especially true for hedge funds -- the survivorship bias in a hedge fund "index" can be 30-40% of the average return.

So what light does

So what light does survivorship bias shine on the oft-repeated claims that because the US is so successful its economic policies must be the best? Nations with less successful policies don't disappear like companies do, of course, but they do change those policies. Wouldn't it seem likely that the most successful economies are the "tournament winners" of your analogy, and not necessarily the best players over time? It certainly seems like nations (e.g. in Asia or South America) that have shown some of the same bipolar distribution shown in your graphs. Maybe such ideas only introduce more variance without affecting (or negatively affecting) median performance.

More on risk Catallarchy

More on risk
Catallarchy tells us about selection bias....

Only The Good (don't) Die

Only The Good (don't) Die Young: Survivorship Bias
Patri Friedman at Catallarchy has some great thoughts on survivorship bias. In the first post, Selection Bias and Risky Strategies, he writes about payoffs to players in poker tournaments. He writes:

Selection Bias and

Selection Bias and Information Security
I read an interesting post about risk strategies and selection bias that made me think about
some short term thinking often seen when investments in information security are deferred.
Patri Friedman discusses poker strategies in light of selection biwe...

Rock climbers (those that

Rock climbers (those that think about the long term, anyway) read the annual report "Accident Reports in North American Mountaineering". The compilers look at all the accident reports they can get their hands on, whether fatal or not. They get reports from climbing partners, rescuers, etc. The report tells you what went wrong and what the climbing party could have done differently that might have made a difference.

As a climber, I read it looking for places to say "I do that! I'll stop doing that." So some people in risky sports are following the example of the airline industry, which seems to investigate all accidents and look for correlations and counter indications to common practices. should someone be doing the same thing for closed funds? If the strategy is just seeking high variance, we won't learn much that way.

"Someone" is doing

"Someone" is doing something, Mr market is busy punishing the stupid and the risky behaviors of individuals, as it should be. Either they don't survive or they learn and adapt.... Or they start a philosophy that rationalizes grinding money out of the successful, like utilitarianism.

Platypus - Whether or not

Platypus - Whether or not success indicates luck vs. skill depends on how much luck and skill affect success. Given the high correlation of characteristics which the US possesses with wealth in general, I think our success is a matter of skill, not luck. If the correlations were low, and economic growth was essentially random, then it would be more evidence of luck than skill.

These correlations are valid because, as you point out, failed nations don't disappear, so we can calculate them without survivorship bias. If we calculated a positive correlation between playing poker like a maniac and winning a tournament by tracking every maniac who entered, that would be evidence that maniacal playing actually works.

Nyet Ya Koshka -

Nyet Ya Koshka - funny!

Chris - I was not trying to implicate rock climbing as risky, but just to come up with an activity where if people died, we wouldn't know how, which is actually kind of difficult in the modern world. Sounds liek it was a bad example. Nyet's is better :razz:

"Fooled by Randomness: The

"Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets" by Nassim Taleb is a brilliantly written exposition of this and other common statistical errors.

Platypus - Which nations

Platypus -

Which nations have tried to copy the US, and how consistently did they do so? I was in Asia during the 1990s SE Asian boom and during at least the beginning of the flu, teaching about financial markets. Once countries such as Malaysia, Thailand, etc. started to take off, they largely stopped trying to learn from other countries. The thinking was that borrowing in foreign currencies without hedging, allowing banks to build up too many bad loans, having little transparency in the stock market, etc. might be bad for other regions but that clearly everything they were doing was right, probably because they had "Asian values". This is human nature - reforms come after disasters or collapses, although they would be easier and more effective in good times, because in good times no one wants to rock the boat.

Countries shouldn't blindly imitate the US, but they should work to establish and adopt international best practices. If a country adopts one or two of the easier policies and doesn't find that they're a magic fix, does that mean that they should ignore international experience and go their own way?

And as for what works and doesn't work for countries, at least we can be pretty sure that communism doesn't merely increase variance - every country that has tried it has declined (one might even say collapsed) economically.

So, Ann, your argument is

So, Ann, your argument is that the "they just didn't do it right" excuse is valid for countries that tried to copy US capitalism, but not for those that tried to implement more leftward alternatives? BTW, no matter how many times people try to portray anything that's not identical in spirit to the kind of economy/market the US has as communist, it remains an inaccurate portrayal. A red herring plus a strawman does not add up to a compelling argument.