Survivorship bias · Lablypedia

Survivorship bias may lead to overly optimistic beliefs because multiple failures are overlooked, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than just a coincidence, as in correlation "proves" causality.

As a general experimental flaw

The parapsychology researcher Joseph Banks Rhine believed he had identified the few individuals from hundreds of potential subjects who had powers of extra-sensory perception (ESP). His calculations were based on the improbability of these few subjects guessing the Zener cards shown to a partner by chance. A major criticism that surfaced against his calculations was the possibility of unconscious survivorship bias in subject selections. He was accused of failing to take into account the large effective size of his sample; that is, all the subjects he rejected as not being "strong telepaths" because they had failed at an earlier testing stage. Had he done this, he might have seen that from the large sample, one or two individuals would probably achieve—purely by chance—the track record of success he observed.

Writing about the Rhine case in Fads and Fallacies in the Name of Science, Martin Gardner explained that the experimenters had made such obvious mistakes not out of statistical naivety, but rather as a result of subtly disregarding poorly performing subjects. He said that, even without trickery of any kind, if a large enough sample were taken, there would always be a certain number of subjects who demonstrated improbable success. To illustrate this, he speculates about what would happen if one hundred professors of psychology read Rhine's work and decided to make their own tests; he said that survivor bias would winnow out the typically failed experiments, while encouraging the lucky successes to continue testing.

He postulated that experiments confirming the null hypothesis (i.e., showing no result) would not be reported, but "[e]ventually, one experimenter remains whose subject has made high scores for six or seven successive sessions. Neither experimenter nor subject is aware of the other ninety-nine projects, and so both have a strong delusion that ESP is operating." He concludes: "The experimenter writes an enthusiastic paper, sends it to Rhine, who publishes it in his magazine, and the readers are greatly impressed."

If sufficiently many scientists study a phenomenon, some will find statistically significant results by chance, and these are the experiments submitted for publication. Additionally, papers showing positive results may be more appealing to editors. This problem is known as positive results bias, a type of publication bias. To combat this, some editors now specifically call for the submission of "negative" scientific findings, where "nothing happened".

Survivorship bias is one of the research issues raised in the provocative 2005 paper "Why Most Published Research Findings Are False", which shows that many published medical research papers contain results that cannot be replicated.

One famous example of survivorship bias occurred in a study by Redelmeier and Singh, which was published in the Annals of Internal Medicine and purported to show that Academy Award-winning actors and actresses lived almost four years longer than their less successful peers. The statistical method used to derive this statistically significant difference, however, gave winners an unfair advantage, because it credited winners' years of life before winning toward survival after winning. When the data were reanalyzed using methods that avoided this immortal time bias, the survival advantage was closer to one year and was not statistically significant.

One more chapter is free with a Lably account.

Examples

Finance and economics

In finance, survivorship bias can manifest as a tendency to exclude failed companies from performance studies because they no longer exist. It often causes study results to skew higher because only companies that were successful enough to survive until the end of the period are included. For example, a mutual fund company's selection of funds today will include only those that are successful now. Many losing funds are closed and merged into other funds to hide poor performance. In theory, 70% of extant funds could truthfully claim to have performance in the first quartile of their peers if the peer group includes funds that have closed.

In 1996, Elton, Gruber, and Blake showed that survivorship bias is larger in the small-fund sector than in large mutual funds (presumably because small funds have a high probability of folding). They estimate the size of the bias across the U.S. mutual fund industry as 0.9% per annum, where the bias is defined and measured as "average α for surviving funds minus average α for all funds", where α is the risk-adjusted return over the S&P 500 (this is the standard measure of mutual fund out-performance).

Additionally, in the financial field, survivorship bias is the use of a current index membership set rather than using the actual constituent changes over time. Consider a backtest to 1990 to find the average performance (total return) of S&P 500 members who have paid dividends within the previous year. To use the current 500 members only and create a historical equity line of the total return of the companies that met the criteria would add survivorship bias to the results. S&P maintains an index of healthy companies, removing companies that no longer meet its criteria as a representative of the large-cap U.S. stock market. Companies that had healthy growth on their way to inclusion in the S&P 500 would be counted as if they were in the index during that growth period, which they were not. Instead, there may have been another company in the index that was losing market capitalization and was destined for the S&P 600 Small-cap Index, which was later removed and would not be counted in the results. Using the actual membership of the index and applying entry and exit dates to gain the appropriate return during inclusion in the index would allow for a bias-free output.

Business

Michael Shermer in Scientific American and Larry Smith of the University of Waterloo have described how advice about commercial success distorts perceptions of it by ignoring all of the businesses and college dropouts that failed. Journalist and author David McRaney observes that the "advice business is a monopoly run by survivors. When something becomes a non-survivor, it is either completely eliminated, or whatever voice it has is muted to zero". Alec Liu wrote in Vice that "for every Mark Zuckerberg, there's thousands of also-rans, who had parties no one ever attended, obsolete before we ever knew they existed."

In his book The Black Swan, financial writer Nassim Taleb called the data obscured by survivorship bias "silent evidence".

History

Diagoras of Melos was asked concerning paintings of those who had escaped shipwreck: "Look, you who think the gods have no care of human things, what do you say to so many persons preserved from death by their especial favour?", to which Diagoras replied: "Why, I say that their pictures are not here who were cast away, who are by much the greater number." A similar story is told about Diogenes of Sinope.

Susan Mumm has described how survival bias leads historians to study organisations that are still in existence more than those that have closed. This means large, successful organisations such as the Women's Institute, which were well organised and still have accessible archives for historians to work from, are studied more than smaller charitable organisations, even though these may have done a great deal of work.

Highly competitive career

Whether it be movie stars, athletes, musicians, or CEOs of multibillion-dollar corporations who dropped out of school, popular media often tells the story of the determined individual who pursues their dreams and beats the odds. There is much less focus on the many people who may be similarly skilled and determined, but fail to ever find success because of factors beyond their control or other (seemingly) random events. There is also a tendency to overlook resources and events that helped enable such success, which those who failed didn't have.

For example, a 2013 study found that 91% of music artists were undiscovered on social media, and just 1.1% were mainstream or mega-sized. The overwhelming majority of failures are not visible to the public eye. Only those who survive the selective pressures of their competitive environment are seen regularly.

Military

During World War II, the statistician Abraham Wald described methods of estimating the vulnerability of various parts of an aircraft based on damage to surviving planes. The Statistical Research Group (SRG) at Columbia University, of which Wald was a member, issued eight memoranda on methods of analyzing data obtained from damaged combat aircraft. Given that there was only data from surviving aircraft, this work directly addresses the issue of survivorship bias. In particular, memo five discusses a method to estimate the vulnerability of various parts of an aircraft based on the damage done to aircraft that had returned from missions. In his comments on his memos, Wald proposed that his conclusions could be used as guide for locating the placement of protective armor. His work is considered seminal in the then-nascent discipline of operational research.

Cats

In a 1987 study, it was reported that cats who fall from fewer than six stories and are still alive have greater injuries than cats who fall from higher than six stories. It has been proposed that this might happen because cats reach terminal velocity after righting themselves at about five stories. After this point they relax, leading to less severe injuries in cats who have fallen from six or more stories. In 1996, The Straight Dope newspaper column proposed that another possible explanation for this phenomenon would be survivorship bias. Cats that die in falls are less likely to be brought to a veterinarian than injured cats, and thus, many of the cats killed in falls from higher buildings are not reported in studies of the subject.

Studies of evolution

Large groups of organisms called clades that survive a long time are subject to various survivorship biases such as the "push of the past", generating the illusion that clades in general tend to originate with a high rate of diversification that then slows through time.

Business law

Survivorship bias can raise truth-in-advertising issues when the success rate advertised for a product or service is measured by reference to a population whose makeup differs from that of the target audience for the advertisement. This is especially important when

the advertisement either fails to disclose the relevant differences between the two populations, or describes them in insufficient detail; and
these differences result from the company's deliberate "pre-screening" of prospective customers to ensure that only customers with traits increasing their likelihood of success are allowed to purchase the product or service, especially when the company's selection procedures or evaluation standards are kept secret; and
the company offering the product or service charges a fee, especially one that is non-refundable or not disclosed in the advertisement, for the privilege of attempting to become a customer.

For example, the advertisements of online dating service eHarmony.com fail a truth in advertising test because they fail the first two prongs and pass the third, when all three must be passed:

they claim a success rate significantly higher than that of competing services while generally not disclosing that the rate is calculated with respect to a viewership subset of individuals who possess traits that increase their likelihood of finding and maintaining relationships and lack traits that pose obstacles to their doing so, and
the company deliberately selects for these traits by administering a lengthy pre-screening process designed to reject prospective customers who lack the former traits or possess the latter ones, but
the company does not charge a fee for administration of its pre-screening test; thus its prospective customers face no "downside risk" other than wasting their time, expending the effort involved in completing the pre-screening process, and suffering disappointment.

Academia

Survivorship bias can distort perception of academic careers because advice and retrospective accounts tend to come from successful researchers who remained in academia, while those who made similar choices but left (or were denied tenure) are less visible. This can make successful career paths appear more reproducible than they actually are.