How to mitigate survivorship bias in life?

In my last article, I defined survivor-ship bias as a judgement error caused when you encounter survivors more often than failures. This makes us over-price the best case scenario and under-price the worst case scenario. We discussed what happens when survivorship bias persists – there is a significant gap between reality and expectation over time and this gap tends to increase frustration, anger, anxiety and desperation. 

Scenario 1

Let’s say you are a solo entrepreneur looking for a tech co-founder to realize your dreams. You start attending tech meetups to identify like minded people – you find someone and decide to start a company with this guy… Do you see survivorship bias in this?

Its like speed dating – you both share a dream and decided to hook up. Neither one knows how the other guy reacts in adversity, how much pain and uncertainty can (s)he endure or how much integrity does the opposite person have. There is a potentially deadly form of survivorship bias at play here – I’ll explain why it is deadly in just a few minutes.

Scenario 2
Let’s look at a second scenario – you have started the company, rolled out a product & made some early sales. You feel the need for a professional sales guy to take selling to the next level. You look to hire a sales head – you get a flood of CV’s. Here is a sample CV.

This lady helped generate $50,000 in sales for her previous company, check.. she led a team of 20 sales people, check. She was nominated as the salesman of the year, check. She has a positive mindset and focuses on team play, check check, check and check. Do you notice survivor ship bias in this? Absolutely.

Every CV that you will ever see is just a proof that massive amount of survivorship bias exists around us.


For all we know, she had absolutely no role to play in converting a sale but helped in the process by doing a few presentation to junior folks. Someone on her team would have pitched to decision maker and got the business. There is no way you can know what her real worth is by reading this paper. That’s why I find CV as an outdated concept for today’s workplace.

Scenario 3
Next, lets look at a third scenario – you are looking to hire a python developer and ask your friend’s help to interview this guy. Your friend asks some pointed questions during the interview on how the candidate maintained his code base, what libraries did he refer to and so on. If a person didn’t do what he claims to have done on his CV, there is a very high likelihood he will be caught right there. In scenario 3, notice how there is a lower risk of survivorship bias.

This is where we touch upon the important concept of a hard skill v/s soft skill.

Hard skills v/s soft skills

If I ask you, what is the probability that a NFL draft player has below average skill level, you will say that it is close to zero. The fact that his name was even considered in the draft means that he must be an above average player with a high degree of skill. Same is the case if I ask you about someone acting in a Broadway play or a pilot flying with 25,000 hours of experience or a surgeon at Harvard medical school. The chance of misjudging a person’s ability because of survivorship bias is very low in areas of hard skill. Hard skills are coding, conducting a surgery, cooking, music, flying and so on..

Now lets look at another bunch of professionals – lets say a fund manager who has demonstrated phenomenal returns in the past 3 years or a sales guy at a company that has tripled sales in past 12 months or a CEO who took a company to an IPO or a sports coach who turned around a struggling team. Would you have the same level of confidence that these people can repeat that performance in future? Was the performance really due to their abilities or due to some external factor or just a pure random event?


If we plot years of experience on X-axis and chance of survivorship bias on the y-axis, we will see that as the years of experience increase, a person with hard skills is less likely to demonstrate survivorship bias. We can predict outcomes with high degree of confidence in such cases.


Now lets look at soft-skills. As years of experience increase, a person with soft skills can demonstrate a higher risk of survivorship bias. So a grey haired advisor who has advised a lot of successful companies and demands a hefty retainer could be a possible outcome of survivorship bias. You possibly end up hiring a white elephant who talks big but delivers nothing on the ground. I’ll talk about how to mitigate this risk in a few minutes.

Areas where there are soft skills at play are potential breeding grounds for survivorship bias.


Why does this happen? If we go back to our first lesson in ‘risk and probability’, you will notice that outcomes delivered by people with soft skills tend to have much fatter tails than those delivered by people with hard skills. Whenever tails become fatter, risk of worst case scenarios increases, and tendency of ignoring worst case & overestimating best case becomes very high.

 
An outcome is a function of many variables – a hard skill is a like a solid shell that wraps around these variables and controls the outcome with a high degree of likelihood. A soft-skill on the other hand is like a porous shell and outcomes are controlled by variables outside of the shell. Soft-skill person tends to believe that he is controlling the outcome by influencing variables under his control, ie within the porous boundary. Reality could just be that during the period of observation, outside variables did not adversely impact the outcome. But that is no guarantee that he can deliver similar performance in future.

Another important point is that not all risks due to survivorship bias are the same – some risks lead to a total ruin while some risks lead to a setback that we can recover from. In our previous scenarios, risk of choosing a wrong co-founder is bigger than risk of choosing a wrong sales head. Choosing a wrong co-founder can fundamentally damage the prospects of success. We need to be extra careful to mitigate survivorship bias when risk of ruin is high.

That takes us to next topic – why does survivorship bias exist in the first place. I attribute it to 2 reasons:

  1. Network effect of success
  2. Our inability to understand randomness

Network effect of success

Network effect is the ability of humans to propagate ideas to a large number of people. If we plot Network effect against Success, we will see that at some level of success, network effects suddenly increase manifold.

Be it word-of-mouth, or Public Relations or Advertisements – people will come to know about the success sooner rather than later. So at this end of the curve, success talks and at the other end of the curve, there are millions of companies that maintain a low profile and aren’t known to people at large.

If you heard about a product or service, there is already a built-in survivorship bias at play.

Causality

Humans dislike randomness – we tend to attribute a reason to everything, be it a success or failure. Even if the event is completely random, our tendency to create a cause for that event often creates a survivorship bias.

For eg If we see a sales track-record for a guy with a good education background and work experience, our mind attributes education and experience as the reason for that track record. In reality, there might be other factors at play such as great marketing team in the background, an early mover advantage for the product, or an established brand with well-laid out sales processes.

Similarly, if we see an investor with a superb track record record who is X-Goldman and a Wharton graduate, we immediately attribute his success to a brilliant understanding of markets. Actual reason might well be that the current economic cycle suits his style of investment which might not be the case in future. 


Tendency to look for a cause for success even if there exists none can trigger survivorship bias

How to mitigate survivorship bias

  1. Make more observations over time

Biggest antidote of survivorship-bias is time. If you observe something long enough, then you will be less prone to judgement errors due to survivorship bias. If you see a bulls-eye on a dartboard, wait till the player throws more darts at the board – if you see darts scattered across the board, you can safely assume that bulls-eye was a random event.

2. Reduce exposure to media
Media as an organization, dislikes regular, mundane events and is on a constant lookout for extra-ordinary events. Reduce intake of media and survivorship bias automatically reduces to a large extent. Instead rely more on primary sources such as friends & family, industry experts, word-of-mouth and your own research/experiences for gathering information

3. Burden of proof is on the guy with soft skills

As they say, ” even a broken clock is right twice a day”. When it comes to hiring advisors, consultants, sales people and others with soft skills, always suspect their capability unless proven otherwise. If they are genuinely good, then you can always change your opinion later. But believing in claims & ‘CV material’ upfront can make your a victim of survivorship bias that could be costly for your business.

4. Focus on the process, not the outcome
Last, but not the least, always focus on the process and not the outcome. Simply because, process has lesser randomness than outcome. Whether it is sales, marketing, leadership, investment – base your decisions on the underlying processes rather than the publicly visible outcomes.

Summary

  1. Hard skills exhibit lower risk of survivorship bias over time. On the contrary, soft skills, as experience increases, see an increase in the likelihood of survivorship bias.. 
  2. Network effects make success talkative & failure silent. 
  3. Humans aren’t comfortable with randomness – we tend to attribute a cause to success even if there exists none.
  4. Time is a biggest antidote to survivorship bias. Invest time and observe as many outcomes as possible before deciding future course of action
  5. Discount what you see & hear in media, suspect everyone with soft skills until you see proof otherwise, and focus on process and not the outcomes. Follow these & you should be able to filter a large percentage of survivorship bias that exists across the world.