Understanding Survivorship Bias

Sep 29

survivorship-bias-misconception-myth-fallacy-mountain-success-business

Takeaway Points:

Survivorship bias happens when we only see the successes (survivors) individuals and businesses have, which can lead to a warped perception of success rates.
Social media creates constant survivorship bias because people only share the good and successful parts of their lives in order to make themselves look better and trustworthy, naturally hiding the parts they don’t want us to see.
Most of the time, success takes a lot of effort and failure, but people on the outside don’t see behind the scenes. Don’t be fooled - continue to put in the work if you want to improve!

You thought I was done writing about cognitive biases and logical fallacies after that stint in 2018, huh? Well, you thought wrong!

More recently, I’ve been thinking a lot more about one cognitive bias that I didn’t really cover well enough back then, and want to dig a lot more into - the survivorship bias. I’ve discussed it a bit before, but I haven’t given it the attention that it deserves.

Survivorship bias is a bias which results from the fact that while failures are often hidden and invisible, successes are often very visible and highly publicized. This leads to numerous false assumptions - that successes are actually more common than they really are, and that failure is not a normal and acceptable part of the process.

The Online Coach Problem

Numerous online coaches share success story testimonials in order to help find clients and make sales. This is an accepted standard - I certainly do it myself! - but it can be particularly deceptive when a coach isn’t actually a very good coach, but still manages to have a large audience.

Let’s take two hypothetical business coaches - let’s say that coach one has an 80% success rate (however you measure that), while coach 2 has a 30% success rate. Naturally, you would think that it would be obvious that coach 1 is the superior coach. If they were both equally good at publicizing themselves, this might be the case.

Now let’s say that coach 2 is way better at building an online audience, despite the fact that they’re a bad coach. Let’s say that coach 1 works with 100 clients in a year, while coach 2 works with 500. Now, at the end of the year, coach 1 has 80 successful clients, and can use those success stories to bolster their business. But at the same time, coach 2, despite having a much worse success rate, still has 150 successful clients - twice as many as coach 1. This coach can create an even greater impression of success - they have twice as many success stories! - despite the fact that they are, objectively, a pretty bad coach.

The problem here is that the coach in either case is naturally only going to selectively choose those “successes” to display in order to promote their business - obviously, you wouldn’t promote failed clients because this makes you look bad. But if you’re not promoting those failures in any way, the end result is that you create a world in which it looks like all your clients succeed, even when they don’t.

People could independently try to do some research to try and figure out your success rate, but this would be incredibly hard - after all, personal contact information for a coach’s clients would be private data, so you’d have no way of finding them unless you managed to stumble into someone who was willing to talk about their experiences. There are plenty of forums and social media sites where folks can discuss their experiences with coaches, but this often doesn’t happen in any comprehensive or coordinated way.

The end result, is that crappy coaches can often get by, despite being pretty objectively bad at their jobs, simply because their failures are not immediately apparent. One can imagine that it would be a very different world if every coach, for example, had to publicly display a chart of all the clients they had worked with, how long they worked together, how successful the relationship was, and so on. In such a situation, coach 2 would clearly be an inferior coach, and coach 1 would start to shine.

This lack of transparency means that, at the end of the day, it’s impossible to really assess who’s good and who’s not. Much in the same way that you can look up a business on google and find that it’s star rating may give you a rough idea of how good they are, but you can’t tell who might just be biased customers with an agenda - we don’t have the kind of clarity or concentration of data required to make a smart decision about how trustworthy or effective some coach on the internet will be. Coaches are self-incentivized to hide their failures and publicize their successes, and the end result is that those few “survivors” (business successes) have an outsized ability to convince us of their value.

This problem is especially relevant when it comes to influencers, who often rely on their social media prowess, their ability to show off their cool lives, etc. to continue growing their client base despite how terrible they might actually be at coaching anybody, or to continue selling products that don’t actually do much of anything useful.

The Stock Market Problem

Let’s say that I want to establish myself as a stock market genius, find a ton of clients, and make a ton of money. What’s the easiest way to do this?

It’s really quite simple. You get a mailing list of sufficient size, let’s say 20,000 people. You take that list, and you send everyone on that list a letter that says that you’ve got some hot new stock tip related to a certain stock. Because you’re so generous, you’re going to tell them this information for free, and they can make a lot of money trusting in your advice. You have to be energetic, and you swear on your life that this information is correct - repeatedly emphasize that they can trust you, you know some trick that they don’t, etc.

But here’s the real trick - you tell half of the people that the stock of choice is going to go UP, and you tell the other half of the people that the stock is going to go DOWN, 10,000 people in each group. You don’t have to be correct, because it doesn’t matter - the people who received the incorrect letter may not invest much money in your prediction, may not even open the letter, may not even care when you turn out to be wrong. They’re certainly not going to impulsively and unexpectedly decide to seek out every other person you’ve sent mail to in an attempt to band against you - that’s an absurdly impossible waste of energy. And you don’t have to care about them either - you just stop mailing them after that.

So now you’ve got a list of 10,000 people who WERE given correct information, and to whom you look like a success. So you repeat the process again, sending half of them a positive letter, and half of them a negative one, about some certain stock. They saw that you were right the first time, so they’re more likely to open the letter, more likely to invest money in your prediction, to trust you, and so on. Just like the first round, you ignore the people who got bad information, and now you’ve got a list of 5,000 people who have seen you, not once, but TWICE predict the future accurately. If you wanted, you could probably repeat this process one or two more times just to make them even more convinced of your skill.

Either way, when you’ve got a sufficient quantity of people convinced of your skill, you now move in for the kill - time to make some money. You offer them some product, some service, or whatever else. Since you’re a stock broker, you’ll sell your stock broker services - and people will trust you because you’ve already given them multiple accurate predictions, just enough that it doesn’t seem like chance.

This, of course, is weaponized survivorship bias. The fact that the people given bad information cannot communicate with the people given good information is exactly the point - all that matters is that some percentage of the people given information succeeded, and that you have the ability to scale this effect to as many people as necessary to make money for your business. Some of the people given bad information (especially after a few rounds of receiving good info) may then choose to act on it personally, and if so, may potentially lose quite a lot of their own money - but that doesn’t matter.

No matter how much hate mail you receive for this process, that doesn’t impact the fact that you’ve got plenty of happy clients. So long as you provide a disclaimer (“information isn’t financial advice, you should be aware that the stock market can go up or down, etc.”), you’re golden.

The Airplane Problem

There’s a graphic going around twitter a lot these days, which is itself a perfect demonstration of the survivorship bias problem. This is the image:

By Martin Grandjean (vector), McGeddon (picture), Cameron Moll (concept) - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=102017718

Here’s the story behind it.

In World War II, research was undertaken in order to determine how to make planes more resistant to being shot down. When they took all the planes that had returned from combat and noted where they had received damage, the above image was what they produced. How should they then reinforce the planes, based on the above image?

The initial conclusion might seem pretty intuitive - obviously, we want to reinforce those areas that tend to take the most damage, which would increase the survivability of the planes - in short, adding armor to all those areas covered in red dots.

However, it turns out that this intuition isn’t just incorrect - it’s disastrously so. The problem is that, as in the other problems above, we’re focusing on the “successes” in this research (the planes that returned from combat) and ignoring the failures (the planes that took critical damage and were unable to return from combat). The above map is not a map of where a plane needs to be more resistant to bullet fire - it’s a map of everywhere on a plane that can safely take bullet fire and return to base.

In short, instead, what we want is to armor up those areas which are NOT represented in the above graphic, because those are likely the areas that, if they take damage, will result in a downing of the plane. Naturally, those areas include crucial elements like the cockpit (where the pilot is located) and the engines - and a hit to any of these could immediately down a plane, while a plane can potentially take a few holes in its less sensitive wings without immediate issue.

source: https://www.countbayesie.com/blog/2020/11/5/survivorship-bias-in-house-hunting-a-practical-modeling-example-using-jax

There’s a similar story from World War I, after the introduction of the Brodie Helmet - the first helmet designed to protect against shrapnel and modern weaponry. Before that, most soldiers would go into combat with little to no protection against modern weaponry, and this meant that a single piece of shrapnel could mean instant death.

So, the helmet was introduced - and all of a sudden, there was a rapid increase in the number of admissions of patients with severe head injuries. This was initially considered to be a potential flaw of the helmet’s design, when it was actually just proof that the helmet was working as intended - because these soldiers were now protected, shrapnel was now far less lethal. At the same time this meant that many potential deaths were converted into serious injuries instead. Still, this was a huge success, because it meant that formerly instantly lethal injuries now had a chance of being treated and recovered from, even if it also meant an influx of head injuries to field hospitals.

How To Combat Survivorship Bias

Survivorship bias occurs essentially in any statistical situation in which successes are far more visible than failures - and this applies to a lot of our personal efforts.

It’s easy to open the Instagram discovery tab and look at what seems like a constant progression of people who seem to “have it all” - highly viral posters who have great bodies, flaunt their wealth, and so on - when the reality is that these influencers make up only a very small percentage of all people on earth. There are numerous more exercisers, for example, who post regularly and try their hardest to develop their physiques, but have just a few hundred followers on instagram and never reach much of anybody.

Likewise, the number of successful people who build a business is necessarily a small fraction of the people who attempt to do so, and very often it has a lot more to do with the luck of being in the right place to capitalize on your efforts than anything to do with skill or hard work. Just because someone has built a successful business once, for example, doesn’t mean that they’ll be a good “business coach” to the next person who comes along. That won’t stop them, of course, from using money to advertise their prowess on social media, and looking for clients to mentor into being the next big success.

Social media is a huge problem, because by its nature, it’s designed to create survivorship bias at all times. Posts which are highly viral get promoted and have a reach far beyond their initial circle, while most people spend all day tapping away at their social media accounts without reaching much more than a small percentage of the people who follow them at any given time. Virality is, by definition, a kind of survivorship bias, and distorts the way that we receive information online towards only the most attention-seeking kinds of content. This is a global problem created by the nature of social media itself, and partially explains the “fake news” epidemic - our feeds are algorithmically curated by virality, and this has a huge effect on the kinds of information we tend to see.

Another major impact of survivorship bias, is that people tend to always think that major accomplishments appear to be easier to accomplish than they really are, in part because people don’t publicize or talk about the hardships of that accomplishment, only the success.

Very often, you might see someone with a successful youtube channel, for example, and see how quickly they blew up, and you might think that they’re just some kind of talented genius when it comes to this sort of thing. But then you dig deeper, and you realize that their channel was secretly toiling away in obscurity for years beforehand, developing the skills required in order to actually start to reach a wider audience. Or you find out that they had another channel beforehand, worked in a similar industry, received a lot of money from a relative in order to get started, or whatever else. Very often, success is NOT fast, and occurs very slowly in obscurity over long periods - we just don’t see it until it actually happens.

This can become discouraging in our own efforts - we put in a bit of practice, it’s not immediately turning out well for us, it does seem to turn out so immediately well for everyone else, so we get frustrated and quit. Again, this is worse when it comes to social media, which is explicitly designed to amplify these tendencies.

As always, I recommend learning to ignore, or even refuse to expose yourself to, most of the messages you find on social media. Social media tends to be bad for our self-perceptions of ourselves, and flooding yourself with images of highly successful people who seem to have everything you want, is only going to distract and discourage you from putting in the effort it takes to really succeed.

About Adam Fisher

Adam is an experienced fitness coach and blogger who's been blogging and coaching since 2012, and lifting since 2006. He's written for numerous major health publications, including Personal Trainer Development Center, T-Nation, Bodybuilding.com, Fitocracy, and Juggernaut Training Systems.

During that time he has coached hundreds of individuals of all levels of fitness, including competitive powerlifters and older exercisers regaining the strength to walk up a flight of stairs. His own training revolves around bodybuilding and powerlifting, in which he’s competed.

Adam writes about fitness, health, science, philosophy, personal finance, self-improvement, productivity, the good life, and everything else that interests him. When he's not writing or lifting, he's usually hanging out with his cats or feeding his video game addiction.

Follow Adam on Facebook or Twitter, or subscribe to our mailing list, if you liked this post and want to say hello!

Enjoy this post? Share the gains!

Share on Facebook

Share on Twitter