Randomness-Driven Recruiting
05 Jul 2016
Hiring is hard and everyone screws it up all the time. Often it feels like the best an interviewer can hope for is to eliminate bias and prejudice and at least make the same mistakes for everyone. However (explanation and caveats to follow), the opposite of bias is not consistency. You can be unprejudiced whilst still being erratic. And when making decisions in a system that is hard to measure and is made up of a small number of events (like, say, hiring), noise and volatility work for you, not against.
In this post I will show how you can use a random number generator to de-homogenize your hiring mistakes. You will find out how to build a stronger team without having to actually get any better at evaluating potential hires. You will see the magic of Randomness-Driven Recruiting.
Problem
Your company makes a lot of mistakes when deciding who to employ. You say yes to dullards, no to geniuses, and everyone has to spend a lot of time dealing with the consequences. If this does not describe you then congratulations! You should stop reading, self-publish an eBook about your recruiting techniques and sell it for $100,000 per word.
Someone trying to hire a new CEO or VP Engineering is in the unenviable position of having to be extremely correct all the time. Randomness-Driven Recruiting will not help them. Instead, let us consider “AppCo”, a completely fictitious company that nonetheless has in its possession a solid team of 50 engineers. AppCo want to expand their operations by hiring more of what they already have. They have set a bar of competency, and are trying to find more people who clear it. As we know, they will undoubtedly turn away many people who are so far above the bar that they regularly whiteboard architecture with God himself, and will hire plenty of others who faceplant directly into the bar but are able to hide their missing teeth and successfully mumble through an implementation of quick-sort. A sensible interview process that helps people show their best self will certainly help, but no amount of take-home tests will eradicate the errors completely.
A team of AppCo’s size can probably absorb a reasonable number of mistakes. They can all make up for each other’s deficiencies. Maybe Steve is much less comfortable with scaling infrastructure than he led his interviewers to believe, but that’s fine because Sally can do this in her sleep. She is, however, secretly terrible at programming in Ruby, but fortunately Barry has worked in Ruby for years. Of course, he has some very exotic flaws of his own. AppCo will keep moving ever onward, as long as the hiring mistakes they make are sufficiently varied. But if they start making the same blunders over and over again, they will start to creak and wobble.
Diversity
This sounds a lot like workplace diversity, and it is certainly related. But what we are considering is actually a very different, more impersonal kind of diversity than you may be used to. Consider AppCo’s rival, DoucheCo. DoucheCo’s goal is to build the least diverse team that they possibly can. They have found a demographic and way of thinking that they like, and they want to stick with it. Whereas AppCo’s ideal candidate is “smart, conscientious, hygienic”, DoucheCo’s is more like “middle-class, good handshake, our sort of chap.”
AppCo sees diversity as a question of good business. Teams with a large number of different viewpoints and backgrounds come up with more ideas, are more likely to question bad assumptions, and just seem to be better. Conveniently, you can get yourself some of these different viewpoints and backgrounds by hiring people from different places who don’t look like everyone else on your team. On the other hand, DoucheCo believes that these considerations do not apply to them, and they sure as hell don’t care about any ethical implications.
This is where we veer sharply off the road and leave what is normally meant by “diversity” a long way behind. Let us suppose that DoucheCo is actually correct, and in their organization, diversity is indeed completely irrelevant. They define a hiring mistake as hiring someone who does not fit their preferred profile of “middle-class, good handshake, our sort of chap.” DoucheCo will still suffer just as much as AppCo from repeatedly making the same kind of mistake, rather than lots of different ones. Suppose that some DoucheCo interviewers make some random, uncorrelated errors that cause them to accidentally hire some poor people, some people with limp handshakes, and some women. This will certainly hamper their ability to do whatever douchey things it is they spend their days doing. But everybody can compensate for each other’s weaknesses (as they define them) and they will stay in business. However, if a systematic error in their screening process causes them to hire a lot of people who all have terrible handshakes, they will have no one who can represent them on the golf course and their organization will wither and die.
We can think about both AppCo and DoucheCo in terms of biodiversity and genetic variation. Both companies have an ideal candidate profile that is adapted for their way of doing business, and both suffer significantly when struck by a common weakness throughout the herd. However, they can resist and maybe even benefit from random mutations in candidate DNA. As we will soon see, they can use a randomness-driven hiring process to exploit this resilience, keep their gene pool large, and inoculate themselves against a company-destroying plague of systematic errors.
Hiring processes
The idea of a hiring process is to understand a candidate’s True Ability and decide whether you would benefit from hiring them. Unfortunately, a candidate’s True Ability goes through a lot of noise- and error-adding translations before it can be used to make a Hire/No Hire decision.
- We start with the candidate’s True Ability
- Then this is translated into a Performance On Interview Day
- Next, an interviewer’s Interpretation Of The Performance
- The interview format will have a Mapping Of Performance To Skills
- The company has to decide Whether It Needs These Skills
- Then a Hire/No Hire Decision can be made
Any of these translations can very easily go cataclysmically wrong, giving AppCo a completely erroneous picture of whether they should hire a candidate.
Each translation has to be made for every single candidate that the company interviews. Importantly, the further down the funnel you go, the more correlated the errors in translation become between different candidates. At the top, it’s a shame for the company when a single candidate over- or under-performs, or when a distracted interviewer fails to notice a glaring error or a flash of brilliance, but it’s not a big, systematic deal. AppCo can even tolerate individual interviewers who are consistently over-impressed by people who use long words or went to the same school as them. As long as they have lots of different interviewers who are all biased in their own, unique ways, mis-translations at this level are likely to cause noise, rather than a systematic skew in one direction.
However, let’s go down the funnel a few steps. The skills that AppCo believes that it is looking for and the ways that its interviews test for these skills have a very consistent, correlated effect on the types of people who are hired. If you state that you are looking for a certain type of person, this is who you will tend to hire. Some of these qualities will, on the face of it, be obvious and unarguable - “we want programmers who can write clean code.” But once you get past obvious truisms, and in particular once you start having to make tradeoffs, everything becomes much more subjective. Do you want people who write clean code slowly or adequate code quickly? What if they got good grades at a good school? Are you sure? How important is it? What if they are a great communicator? Can that make up for a lack of technical experience? How much? Are you being consistent?
Consistency
Suppose that AppCo decides these kinds of tradeoffs are too hard and too random for individual interviewers to answer. They decide that they want to ensure consistency and fairness across all candidates - who doesn’t like consistency and fairness? They establish a small “hiring committee” who will make the final decision on every candidate who passes through their doors. Everyone is evaluated according to the same criteria, in the same way, by the same people.
However, AppCo is now extremely highly leveraged on the opinions and leanings of the people in this committee. Any small biases or mistaken beliefs about what makes a good candidate are replicated across every single interview the company ever does, in exactly the same way. For example, say many AppCo candidates have the majority or all of their experience in Big Companies. If AppCo rigorously evaluate their needs and consciously decide that they don’t want people with a BigCo background then that’s fine, and they should go with their considered opinion. But if they don’t make this evaluation, and the people in their small hiring committee subconsciously happen to dislike people from BigCos, then they will accidentally no longer hire any people with the skills that working in a large team gives you. This will happen without them having ever decided that this was the correct strategy. On the other hand, if they spread their interviews between a large group of interviewers, some will be mistakenly biased against BigCos, some will dislike people who know NodeJS, and others will really hate people from Chicago. These are all sad and silly mistakes to make, but they are uncorrelated and diverse, and AppCo will be able to absorb them.
If AppCo is confident that it can select a hiring committee that will make perfect decisions every single time then it should feel completely comfortable taking this risk. But then it should also get out of whatever business it is currently in and instead sell its services as an infallible hiring oracle, charging $100,000 per decision. If, on the other hand, AppCo is not possessed of any such mystical corporate seers, and is also scared that their interview process may be housing other unidentified, systematic mistakes, then it should step directly through the looking glass and embrace Randomness-Driven Recruiting.
Randomness-Driven Recruiting
As far as I know, every company everywhere assigns candidates a binary thumb pointing up or down. They may be uncertain whether their decision is correct, but they bravely make it anyway. This means that small mistakes in evaluation can tip candidates from a yes to a no or a no to a yes, with no hope for redemption. If AppCo interviewers unconsciously ding everyone whose code is messy but fast down a few rungs, they may accidentally end up with a team containing no one who can prototype a new feature in a small window of time.
If they want to hedge their risk of repeatedly making the same mistake, they should instead assign their candidates a number, P, between 0.0 and 1.0. This number is the probability that the candidate will be hired. They then run their trusty Math.rand to get a random number between 0.0 and 1.0. If it is lower than P, the candidate is hired, otherwise they are told that it was lovely to meet them but they just aren’t a good fit.
This allows AppCo to express not only the candidate profiles that they are looking for, but the degree of certainty with which they believe these profiles are correct. If they are truly certain that an Amish sysadmin would be a bad idea, then they can give them a regretful P of 0.0. On the other hand, suppose that they have only a vague suspicion that they need engineers with startup experience. They can dock the experienced BigCo people a few points without locking themselves out of a skillset that they are not certain that they don’t need.
Currently, everybody uses a version of this system where the only allowed Ps are 0.0 and 1.0. Everything above 0.5 is effectively rounded up, everything below is rounded down. Randomness-Driven Recruiting allows AppCo to exert some degree of deliberate will over who they hire, but only as far as they feel confident. A team that is confident in their ability to choose the exact right people will still come out with a lot of 0.0s and 1.0s. But a team that fears that their interviews may be concealing unidentified, systematic leanings can represent this fear with more 0.7s and 0.2s. If their leanings turn out to be wrong, they will still have some biodiversity to fall back on and keep the group alive.
Effects
Randomness-Driven Recruiting instinctively feels unpleasant and unfair on candidates. Everyone involved in hiring or being hired knows that the process is already fraught with the volatility of inexplicable poor performance, interviewer bias and handshake mis-calibration. But as a candidate, you assume that these elements of chance are the enemy. You assume that the companies evaluating you are striving to eliminate them and get to know The Real You. A candidate who discovers that they are unwittingly being used as a tool to ensure the global health of the herd is likely to feel duped and angry.
But let us suppose a company came clean and said “one of our most important hiring tools is Math.rand.” This still feels very strange, but in a perfect implementation I can’t think of anything morally troubling about it. People have a right to not be deliberately discriminated against, but there is nothing deliberate about well-implemented atmospheric randomness. In practice it might allow unpleasant and very deterministic bias to be disguised as non-deterministic randomness (maybe female programmers turn out to be much less good at rolling dice than the men), but such biases can already easily be covered up without resorting to randomness.
Pause for a moment before you forward this post to your head of HR, recommending that they buy a d20 and send me a large cheque. If you become the only company in your industry to implement Randomness-Driven Recruitment, you should see a shift in the profile of candidates in your pipelines. The 0.2s of this world, who would have never received an offer under the old system, will realize that they now have a shot at being randomly hired. And the 0.7s, who make up a large and valuable part of your team, will decide that they should spend their time applying to companies who give offers to every 0.7 who comes their way. In the perfect information, perfectly fungible and perfectly rational limit, you should only receive applications from people who are between 0.0 and 0.5, along with any perfect 1.0s. No amount of virtual dice can make up for this.
Conclusion
If pushed, I would have to concede that I do not actually believe that any company should play dice with its candidates. Randomness-Driven Recruitment is strange, complex, and just isn’t necessary. Its theoretical benefits are reduced as your underlying interview process gets better and better, and any effort spent calibrating interviewers on how to distinguish a 0.3 from a 0.1 would be much better spent on making better interviews and take-home problems.
On the other hand, you should take care to avoid correlating your errors any more than you already do. Interview rubrics and a defined set of desirable traits and skills are essential, but be wary of centralized hiring committees. Be willing and able to take calculated chances on people you might not always say yes to. They might be a total washout, but they might turn out to be the virtuoso you didn’t know you needed.
Hiring is hard and everyone screws it up all the time. Often it feels like the best an interviewer can hope for is to eliminate bias and prejudice and at least make the same mistakes for everyone. However (explanation and caveats to follow), the opposite of bias is not consistency. You can be unprejudiced whilst still being erratic. And when making decisions in a system that is hard to measure and is made up of a small number of events (like, say, hiring), noise and volatility work for you, not against.
In this post I will show how you can use a random number generator to de-homogenize your hiring mistakes. You will find out how to build a stronger team without having to actually get any better at evaluating potential hires. You will see the magic of Randomness-Driven Recruiting.
Problem
Your company makes a lot of mistakes when deciding who to employ. You say yes to dullards, no to geniuses, and everyone has to spend a lot of time dealing with the consequences. If this does not describe you then congratulations! You should stop reading, self-publish an eBook about your recruiting techniques and sell it for $100,000 per word.
Someone trying to hire a new CEO or VP Engineering is in the unenviable position of having to be extremely correct all the time. Randomness-Driven Recruiting will not help them. Instead, let us consider “AppCo”, a completely fictitious company that nonetheless has in its possession a solid team of 50 engineers. AppCo want to expand their operations by hiring more of what they already have. They have set a bar of competency, and are trying to find more people who clear it. As we know, they will undoubtedly turn away many people who are so far above the bar that they regularly whiteboard architecture with God himself, and will hire plenty of others who faceplant directly into the bar but are able to hide their missing teeth and successfully mumble through an implementation of quick-sort. A sensible interview process that helps people show their best self will certainly help, but no amount of take-home tests will eradicate the errors completely.
A team of AppCo’s size can probably absorb a reasonable number of mistakes. They can all make up for each other’s deficiencies. Maybe Steve is much less comfortable with scaling infrastructure than he led his interviewers to believe, but that’s fine because Sally can do this in her sleep. She is, however, secretly terrible at programming in Ruby, but fortunately Barry has worked in Ruby for years. Of course, he has some very exotic flaws of his own. AppCo will keep moving ever onward, as long as the hiring mistakes they make are sufficiently varied. But if they start making the same blunders over and over again, they will start to creak and wobble.
Diversity
This sounds a lot like workplace diversity, and it is certainly related. But what we are considering is actually a very different, more impersonal kind of diversity than you may be used to. Consider AppCo’s rival, DoucheCo. DoucheCo’s goal is to build the least diverse team that they possibly can. They have found a demographic and way of thinking that they like, and they want to stick with it. Whereas AppCo’s ideal candidate is “smart, conscientious, hygienic”, DoucheCo’s is more like “middle-class, good handshake, our sort of chap.”
AppCo sees diversity as a question of good business. Teams with a large number of different viewpoints and backgrounds come up with more ideas, are more likely to question bad assumptions, and just seem to be better. Conveniently, you can get yourself some of these different viewpoints and backgrounds by hiring people from different places who don’t look like everyone else on your team. On the other hand, DoucheCo believes that these considerations do not apply to them, and they sure as hell don’t care about any ethical implications.
This is where we veer sharply off the road and leave what is normally meant by “diversity” a long way behind. Let us suppose that DoucheCo is actually correct, and in their organization, diversity is indeed completely irrelevant. They define a hiring mistake as hiring someone who does not fit their preferred profile of “middle-class, good handshake, our sort of chap.” DoucheCo will still suffer just as much as AppCo from repeatedly making the same kind of mistake, rather than lots of different ones. Suppose that some DoucheCo interviewers make some random, uncorrelated errors that cause them to accidentally hire some poor people, some people with limp handshakes, and some women. This will certainly hamper their ability to do whatever douchey things it is they spend their days doing. But everybody can compensate for each other’s weaknesses (as they define them) and they will stay in business. However, if a systematic error in their screening process causes them to hire a lot of people who all have terrible handshakes, they will have no one who can represent them on the golf course and their organization will wither and die.
We can think about both AppCo and DoucheCo in terms of biodiversity and genetic variation. Both companies have an ideal candidate profile that is adapted for their way of doing business, and both suffer significantly when struck by a common weakness throughout the herd. However, they can resist and maybe even benefit from random mutations in candidate DNA. As we will soon see, they can use a randomness-driven hiring process to exploit this resilience, keep their gene pool large, and inoculate themselves against a company-destroying plague of systematic errors.
Hiring processes
The idea of a hiring process is to understand a candidate’s True Ability and decide whether you would benefit from hiring them. Unfortunately, a candidate’s True Ability goes through a lot of noise- and error-adding translations before it can be used to make a Hire/No Hire decision.
- We start with the candidate’s True Ability
- Then this is translated into a Performance On Interview Day
- Next, an interviewer’s Interpretation Of The Performance
- The interview format will have a Mapping Of Performance To Skills
- The company has to decide Whether It Needs These Skills
- Then a Hire/No Hire Decision can be made
Any of these translations can very easily go cataclysmically wrong, giving AppCo a completely erroneous picture of whether they should hire a candidate.
Each translation has to be made for every single candidate that the company interviews. Importantly, the further down the funnel you go, the more correlated the errors in translation become between different candidates. At the top, it’s a shame for the company when a single candidate over- or under-performs, or when a distracted interviewer fails to notice a glaring error or a flash of brilliance, but it’s not a big, systematic deal. AppCo can even tolerate individual interviewers who are consistently over-impressed by people who use long words or went to the same school as them. As long as they have lots of different interviewers who are all biased in their own, unique ways, mis-translations at this level are likely to cause noise, rather than a systematic skew in one direction.
However, let’s go down the funnel a few steps. The skills that AppCo believes that it is looking for and the ways that its interviews test for these skills have a very consistent, correlated effect on the types of people who are hired. If you state that you are looking for a certain type of person, this is who you will tend to hire. Some of these qualities will, on the face of it, be obvious and unarguable - “we want programmers who can write clean code.” But once you get past obvious truisms, and in particular once you start having to make tradeoffs, everything becomes much more subjective. Do you want people who write clean code slowly or adequate code quickly? What if they got good grades at a good school? Are you sure? How important is it? What if they are a great communicator? Can that make up for a lack of technical experience? How much? Are you being consistent?
Consistency
Suppose that AppCo decides these kinds of tradeoffs are too hard and too random for individual interviewers to answer. They decide that they want to ensure consistency and fairness across all candidates - who doesn’t like consistency and fairness? They establish a small “hiring committee” who will make the final decision on every candidate who passes through their doors. Everyone is evaluated according to the same criteria, in the same way, by the same people.
However, AppCo is now extremely highly leveraged on the opinions and leanings of the people in this committee. Any small biases or mistaken beliefs about what makes a good candidate are replicated across every single interview the company ever does, in exactly the same way. For example, say many AppCo candidates have the majority or all of their experience in Big Companies. If AppCo rigorously evaluate their needs and consciously decide that they don’t want people with a BigCo background then that’s fine, and they should go with their considered opinion. But if they don’t make this evaluation, and the people in their small hiring committee subconsciously happen to dislike people from BigCos, then they will accidentally no longer hire any people with the skills that working in a large team gives you. This will happen without them having ever decided that this was the correct strategy. On the other hand, if they spread their interviews between a large group of interviewers, some will be mistakenly biased against BigCos, some will dislike people who know NodeJS, and others will really hate people from Chicago. These are all sad and silly mistakes to make, but they are uncorrelated and diverse, and AppCo will be able to absorb them.
If AppCo is confident that it can select a hiring committee that will make perfect decisions every single time then it should feel completely comfortable taking this risk. But then it should also get out of whatever business it is currently in and instead sell its services as an infallible hiring oracle, charging $100,000 per decision. If, on the other hand, AppCo is not possessed of any such mystical corporate seers, and is also scared that their interview process may be housing other unidentified, systematic mistakes, then it should step directly through the looking glass and embrace Randomness-Driven Recruiting.
Randomness-Driven Recruiting
As far as I know, every company everywhere assigns candidates a binary thumb pointing up or down. They may be uncertain whether their decision is correct, but they bravely make it anyway. This means that small mistakes in evaluation can tip candidates from a yes to a no or a no to a yes, with no hope for redemption. If AppCo interviewers unconsciously ding everyone whose code is messy but fast down a few rungs, they may accidentally end up with a team containing no one who can prototype a new feature in a small window of time.
If they want to hedge their risk of repeatedly making the same mistake, they should instead assign their candidates a number, P, between 0.0 and 1.0. This number is the probability that the candidate will be hired. They then run their trusty Math.rand to get a random number between 0.0 and 1.0. If it is lower than P, the candidate is hired, otherwise they are told that it was lovely to meet them but they just aren’t a good fit.
This allows AppCo to express not only the candidate profiles that they are looking for, but the degree of certainty with which they believe these profiles are correct. If they are truly certain that an Amish sysadmin would be a bad idea, then they can give them a regretful P of 0.0. On the other hand, suppose that they have only a vague suspicion that they need engineers with startup experience. They can dock the experienced BigCo people a few points without locking themselves out of a skillset that they are not certain that they don’t need.
Currently, everybody uses a version of this system where the only allowed Ps are 0.0 and 1.0. Everything above 0.5 is effectively rounded up, everything below is rounded down. Randomness-Driven Recruiting allows AppCo to exert some degree of deliberate will over who they hire, but only as far as they feel confident. A team that is confident in their ability to choose the exact right people will still come out with a lot of 0.0s and 1.0s. But a team that fears that their interviews may be concealing unidentified, systematic leanings can represent this fear with more 0.7s and 0.2s. If their leanings turn out to be wrong, they will still have some biodiversity to fall back on and keep the group alive.
Effects
Randomness-Driven Recruiting instinctively feels unpleasant and unfair on candidates. Everyone involved in hiring or being hired knows that the process is already fraught with the volatility of inexplicable poor performance, interviewer bias and handshake mis-calibration. But as a candidate, you assume that these elements of chance are the enemy. You assume that the companies evaluating you are striving to eliminate them and get to know The Real You. A candidate who discovers that they are unwittingly being used as a tool to ensure the global health of the herd is likely to feel duped and angry.
But let us suppose a company came clean and said “one of our most important hiring tools is Math.rand.” This still feels very strange, but in a perfect implementation I can’t think of anything morally troubling about it. People have a right to not be deliberately discriminated against, but there is nothing deliberate about well-implemented atmospheric randomness. In practice it might allow unpleasant and very deterministic bias to be disguised as non-deterministic randomness (maybe female programmers turn out to be much less good at rolling dice than the men), but such biases can already easily be covered up without resorting to randomness.
Pause for a moment before you forward this post to your head of HR, recommending that they buy a d20 and send me a large cheque. If you become the only company in your industry to implement Randomness-Driven Recruitment, you should see a shift in the profile of candidates in your pipelines. The 0.2s of this world, who would have never received an offer under the old system, will realize that they now have a shot at being randomly hired. And the 0.7s, who make up a large and valuable part of your team, will decide that they should spend their time applying to companies who give offers to every 0.7 who comes their way. In the perfect information, perfectly fungible and perfectly rational limit, you should only receive applications from people who are between 0.0 and 0.5, along with any perfect 1.0s. No amount of virtual dice can make up for this.
Conclusion
If pushed, I would have to concede that I do not actually believe that any company should play dice with its candidates. Randomness-Driven Recruitment is strange, complex, and just isn’t necessary. Its theoretical benefits are reduced as your underlying interview process gets better and better, and any effort spent calibrating interviewers on how to distinguish a 0.3 from a 0.1 would be much better spent on making better interviews and take-home problems.
On the other hand, you should take care to avoid correlating your errors any more than you already do. Interview rubrics and a defined set of desirable traits and skills are essential, but be wary of centralized hiring committees. Be willing and able to take calculated chances on people you might not always say yes to. They might be a total washout, but they might turn out to be the virtuoso you didn’t know you needed.