Comment by thethimble
11 years ago
Think of the problem from Google's perspective though. At some point, you have tens of thousands of candidates and you need a system to quantify how good they are. Further, it's reasonable to have false negatives (people you don't hire that should have been hired) but really bad to have false positives (people that you hire that you should not have). Together, these boil down into the de facto whiteboarding interview process.
This has got to be the biggest hiring fallacy I've ever heard. "It's better to reject a good candidate than hire a bad candidate."
That's completely false, and anyone who says that is completely ignorant of Bayesian logic.
Here are some simple numbers.
Suppose that a "good" candidate is a 1-in-100 find. Suppose that a "bad" candidate has a 1% chance of tricking you into hiring them anyway.
Every time you pass on a "good" candidate, that is greater opportunity for a "bad" candidate to trick you into hiring them!
Counterintuitively, if you pass on too many of the "good" candidates, in your overzealousness to reject bad candidates, you're actually INCREASING THE ODDS OF A BAD HIRE.
This is management porn. "It's better to reject a good candidate than hire a bad candidate." It makes the manager feel good. "Wow! That candidate seemed smart, but I rejected him anyway! I'm such a great leader! I make the tough decisions!"
tl;dr summary
Because "good" candidates are rare, every time you pass on a good candidate that increases your odds of making a bad hire! This is simple Bayesian reasoning!
Those are simple numbers, but they don't get at the real issue. People say that because 1 bad hire can have negative effects on the whole team, causing others to get less done and leave. Missing a good hire doesn't poison your team.
But 1 bad hire is easily correctable - you fire the person. You won't know that you missed on the good hires.
I personally have worked with several awesome people that have interviewed with Google, and none of them got hired. They all said the interview process was flat out insulting.
Interestingly enough, most of them ended up at Facebook.
10 replies →
The good and bad the op is talking about is different. If the attitude is not the best, you don't hire anyway.
If the candidate cannot invert a binary tree and gets mostly there, and has demonstrable real world software, it's a different thing.
1 reply →
>People say that because 1 bad hire can have negative effects on the whole team, causing others to get less done and leave. Missing a good hire doesn't poison your team.
That doesn't disprove what he said. In fact it strengthens it and makes missing a good hire even worse, because, as he said missing good hires -> MORE change of bad hires (that can "poison your team").
In other words, it's not a choice: "I'm better of rejecting a good hire than getting a bad one, because a bad one could poison the team".
By rejecting good ones, who're actually getting MORE bad ones. So it should be corrected to:
"I'm better of NOT rejecting good hires, or else I'll get more bad hires and poison my team".
This is akin to banning food because people might get a stomach ache, and this is bad for the team.
> Counterintuitively, if you pass on too many of the "good" candidates, in your overzealousness to reject bad candidates, you're actually INCREASING THE ODDS OF A BAD HIRE.
Yes, but what hiring managers are doing isn't passing over candidates that they know are good, they're passing over candidates that they're not very sure about.
> Because "good" candidates are rare, every time you pass on a good candidate that increases your odds of making a bad hire! This is simple Bayesian reasoning!
Similarly, of course this is true, but again hiring managers aren't passing up candidates that they know to be good, they're passing up on candidates that they're not sure about, whom the acknowledge could be good.
Under certain conditions, every time you pass on a candidate you're not sure about, you decrease your chance of making a bad hire. The conditions are that increasing hiring standards must weed out more bad candidates than good candidates.
Let's work with your model. Suppose 1% of candidates coming for an interview are good and 99% are bad.
Hiring strategy A manages to hire all good candidates it interviews, and 1% of the bad candidates it interviews. End result: for every 100 people you interview, you get 1 good candidate and 1 bad candidate.
Hiring strategy B is more conservative and hires 50% of all good candidates it interviews, and 0.1% of the bad candidates it interviews. End result: for every 100 people you interview, you get 0.5 good candidates and 0.1 bad candidates.
The claim is something like, the team produced by strategy B is better than the team produced by strategy A, even though team B has less good candidates than team B.
The slogan represents the opposite. You hire only 10% of the good candidates but the odds of hiring a bad candidate are cut to only 0.5%.
3 replies →
This is not about numbers.
A bad hire can easily cause more friction than if nobody was hired at all. Net effect for the business is negative. You're also probably severely overestimating the likelihood of a bad hire going through.
Bad hires can be fired. Bad rejections poison the well for hiring the good candidates who never apply because of hearing about all the bad rejections.
2 replies →
You can fire a bad hire. You can't hire someone you already rejected because a few weeks later they already have a job.
8 replies →
Doesn't understand bayesian logic! Doesn't understand risk management!
7 replies →
Not really. The population is large enough that you're treating each potential hire as a random variable. Given the amount of possible good new hires the effect of passing on one actual good hire has negligible effect on the next potential hire.
The theory behind the "raise the bar" argument is that obviously good hires are easy to spot (this is the part most people reasonably take issue with), so people who do not pass that bar of "obviously good" aren't worth taking a risk on. The conclusion is statistically reasonable if you accept the premise (which, again, you probably shouldn't).
> The population is large enough that you're treating each potential hire as a random variable.
This is only true for new grads. The population of engineers with ten years of appropriate corporate experience (that match the technology survivor bias) and at the ~12 locations google can use is tiny.
When I look at my old corporation, only 3 people (of 500+ I would be aware of) were hired by google. One of them is genuinely great, but I doubt he passed the current process as a blind hire. The other two I wouldn't want to work with, but I bet they passed this process precisely because the right kind of focus. (I.e. everything people around them didn't know was their priority and since we normally focus on what's important to completing our project, none of that was so important..)
I was very interested in google when I was under the impression that (without experiencing menlo park on a daily basis,) I could either walk away with over $500k (gross) in 2- years, enjoy working there for 5 years, and/or work with the best >experienced< engineers in our field. When glassdoor and my own linkedin network revealed my best perspective into their setup, blind applying to them is now just a good filter to test my new age employability skills. So, now they've built themselves a new problem of true positive false positives who are just along for the ride.
The hiring market is already NP Hard for both sides when you are genuinely trying to get a good enough fit for yourself rather than the best deal on the market. If you try to misuse the other side they will see less ethical constraints in using the resources you'll have to put in.
How many developers genuinely want a job they can't do that they will have to be awkwardly fired from in six months and explain at every future interview?
The difficult thing with hiring bad candidates vs not hiring bad candidates is that you get to experience the bad hire every single day, so it is quite apparent that you made a mistake. With not hiring a good candidate, you don't really know what you were missing. You may just assume that the best applicant you reject is no better than your average employee. But there is a chance that that person would bring something to the table so unexpected that you can't even imagine the world that would have been had you hired them.
Obligatory reference to tptacek's post at http://sockpuppet.org/blog/2015/03/06/the-hiring-post/ -- see the end of chapter 2 for an amazing example
Apparently you hadn't been recruiting anyone in Europe. Once you've hired a bad candidate, it is very hard to get rid of it (borderline impossible in Scandinavia, for example).
You're conclusion rests on the numbers you start with, those aren't a given.
I understand your point, and I'm not trying to be a troll.
Humans are more complex than that. I don't think you can assume that candidates will perform the same all the time. Sometimes an excellent candidate can perform badly for multiple reasons (e.g. nervousness, poor preparation, bad interviewer, personal problems, etc).
It seems to me, that rejecting a good candidate, and have him/her interview again after some time, if that candidate was a 'good-hire', then it would increase the chance of hiring him/her, since it is most likely they will prepare better, and know what to expect.
Why would anyone who has a job waste a vacation day to interview at a place that previously rejected him?
If your flawed process rejected a good candidate the first time around, what makes you think the same flawed process won't reject them a second time?
3 replies →
If only whiteboarding interview processes actually weeded out false positives.
In practice, they select for people with good memorization skills. If you can remember the details on a ten dozen different algorithms and data structures, you can pass one of these without having a single lick of creativity or skill.
I say this as an employee who has worked alongside many unskilled drones who made it past the algorithmic interview process.
I agree.
I recently interviewed for a security engineering position at a startup, and while I got offered the job, the interview was quite silly.
The first thing I was asked to do was to write a lisp interpreter. It was a pretty trivial task, but it left me scratching my head since not only did it not tell them about whether I would be qualified at all, but I had explicitly avoided writing parsing code since I found ANTLR during undergrad.
In the end I wrote a basic stream abstraction and wrote my lisp interpreter on top of it with no real difficulty, but it was a completely stupid question. I told them on the third interview that they weren't asking me anything relevant and I still got another question later on about how to iterate over a tree in a specific order.
Part of the issue was that most of my interviewers were fresh out of college, it was literally their first job and they had no idea what to ask besides the kinds of questions they had been asked in their own interviews.
That last sentence is telling. Would you even consider working there if they had no idea what to do? I have worked at established companies where there is clearly nobody at the helm and it is frustrating.
1 reply →
That's a great point.
Infatuation with "coding competitions" and high flux intelligence tasks is often an anti signal.
Sure, what they demonstrate looks good at face value, but unless you attach a micromanager over their tasks, they'll constantly deviate to the new-shiny every week instead of staying on task for the long haul.
I know even YC has been bitten a few times by accepting people who are top-performers in very narrow contexts (e.g. people obsessed with winning coding competitions every single weekend) only to see they can't do anything coherent for more than a week at a time.
Just because someone is great in a narrow context doesn't mean they can carry that over the months and years needed to deeply iterate (and suffer through the low points) on successful projects.
I'd generalize your point a little more - for a given level of life success, intelligence will anti-correlate with other hire-ability skills like work ethic, diligence, attendance, emotional stability, professionalism, etc.
Basically, imagine that there's two ways to graduate college with a CS degree - you're either the kind of person who's smart enough to get things if you work your ass off for them, or you're a genius and you coast through. If you're a genius and you work your ass off, you get a doctorate or found a startup, and aren't in the candidate pool. So you wind up with a choice about what to compromise on: work ethic, or smarts.
Ideally, you compromise on whatever makes the least difference to your organization, compared to your competitors. Hiring smart/motivated people out of non-traditional backgrounds is a great option for this.
2 replies →
Which is why I hate HackerRank being used as a screening tool.
That's not what the interview process is like at all.
They're more interested in how you approach real world issues (the questions I got asked were conceivably real-life issues a company like Google would face with its products). If you can solve that issue by applying efficient and well understood data structures and algorithms, then that indicates you understand the problem space and solutions that may apply.
It's not like they get you in a room and ask you to draw a linked list or a binary tree.
I'll be starting at Google in a week and a half. I can't go into specifics, but I can say this:
I had two phone interviews - the first had quite a few questions about my prior experience (over a decade) and how to approach real-world problems from a high-level, including some that I had worked on at my current position. The second was primarily purely technical, but did have some more theoretical/project-management-y type of questions. In neither one was I required to write any code.
On-site: one interview was entirely whiteboard-coding (in a collaborative style), on a set of related generic problems. Next was one with a set of problems related to my experience. Third was almost entirely telling war stories. Last two were some generic graph-theory-type problems and some simple problems involving data structures relevant to the my area of experience.
As I hadn't interviewed for anything in 11 years, I did spend some time brushing up on my basic algorithmic theory, and essentially followed Steve Yegge's suggestions. I honestly think I probably could have done the vast majority of it without any studying, but spending time practicing solving problems on paper (with a countdown-timer as an artificial pressure-inducing device) certainly did help me, I think, in the on-site. No amount of memorization could have helped with most of it, aside from basic knowledge that anyone who's taken Data Structures should know.
I didn't find the process to be insulting at all; it was an enjoyable challenge. Everyone I talked with on the engineering side of things I can only say the best about, and so far as I can tell the team I'll be with has some really great people. I am a more senior developer, so perhaps I got a different experience than some.
2 replies →
My experience wasn't a real life type one more academic instead. My interviewer was really late and refused to let me finish my data structure implementation to optimize a single method instead. When we finished it was time for him to go. He wouldn't let me go back to finish my data structure as he didn't have the time.
I was later declined. Had a couple of friends at google ask the interviewer. He said I was declined for not finishing my data structure.
Their interviewing process was quite frustrating. I'm fine with getting declined but not under those circumstances.
3 replies →
The important part of the "realistic" process is the part where you go from "the crawler's indexing is too slow" to "it might be because it allocates a lot of little bits of memory; it needs to first generate a BST of outbound link weights, and then functionally invert that into a second graph with a bunch of inbound link weights, and then inject all that into the PageRank matrix. You could probably save some time (though at the cost of memory-safety) by inverting the first BST in-place."
Once you get to that point, though, if they asked me "and how would you invert the tree in-place?" I'd answer "well, first I'd make sure we aren't just using a tree structure that already knows how to invert itself, or could easily be extended to know how. Then, assuming that didn't pan out, I'd google 'in-place invert binary tree' and read three or four solutions that already exist to burn the semantic structure of what I'm about to write into my head so that I don't subtly screw it up and have to fiddle with it for the next hour when I could be coding something else. Then I code it."
And that should be enough.
One of my interviewers at Google asked me to implement a balanced tree insert on the whiteboard in C. After doing so he immediately told me it wouldn't work. I searched futilely for a minute or two to find the problem before he condescendingly pointed out I had left out a semicolon on one of my statements...
Another in the same interview got in my face (this should be easy) when he thought it was taking me too long too multiply two numbers like 637 and 41 during an estimating problem. I understand the point of most of these questions is quick and dirty estimation (600 x 40), but seriously...
4 replies →
> It's not like they get you in a room and ask you to draw a linked list or a binary tree.
And yet the OP was turned down for not being able to invert a binary tree (based on limited information, etc etc.)? What do you feel that is, if not a "whiteboard this algorithm" question?
As for many of the other companies who try and emulate Google - this is exactly what they do.
5 replies →
That said, I recently heard about a Google interview where the interviewer asked the question and left the room for 30 minutes while the candidate white boarded.
Obviously he didn't give 1 shit about "how you approach the real world issues".
They literally had me do a deterministic finite automaton problem.
6 replies →
I think what Google tries to find are the engineers with creativity, skill and ability to work intelligently at a problem which they haven't seen before.
It's important to know how a person approaches a hard new problem and how easily or not they give up. I'd guess that a person who didn't solve a hard problem, but got close enough by displaying creativity, curiousness and a solid thought process might have an edge over a person that implemented a familiar algorithm with ease but got a little flustered, and didn't show much creativity or inquisitiveness, when challenged with something new.
I think you want to think that way, that whiteboard interviews have a shred of competency about them. I bet you dollars to doughnuts though 9/10 times a company will hire someone who can regurgitate algorithms rather than an eccentric who can't get them right but thinks really cleverly.
4 replies →
No, it doesn't.
Say I pass the invert a tree question. Does that prove I am good? That I can design a product, listen to customer requirements, come up with ideas that elude others, read a research paper and turn it into something commercial, solder a circuit board, make a schedule, mentor junior engineers, write documentation? No. All inverting a tree tells them is I studied trees recently, and/or I'm at least superficially clever.
Google lets go of plenty of people. They aren't making perfect hires.
They have publicly admitted there's no correlation between their hiring process and outcomes. I'm not even clear [why] we're still debating this.
Other than to wonder why they still haven't done anything about it. (I interviewed there ~8 years ago, pre Android when they were looking for deep mobile expertise and got flunked out on a similar problem. I am not at all interested in working for them now)
To show that there is a correlation you have to quantify both hiring process evaluation and job performance. Since both are elusive quantities that are hard to quantify, showing there is no correlation may just mean that measurements are done poorly. E.g. when job success is measured by boss satisfaction.
None of the things you describe are part of the job for a rank and file bigco engineer. They already have filled all the leadership roles.
> you need a system to quantify how good they are
But do you? How catastrophical would it be if Google accidentally hired not the top 0.001% but instead somebody from top 0.01%? I mean, same people that build everything else around us and the world still didn't collapse yet? I get it, you have to have standards, and you don't want to waste time and money on somebody that can't pass fizzbuzz test. But once you've moved past that - do you need to obsess so hard on ranking and quantifying it? There are probably some positions in Google that require people to invert binary trees (speaking figuratively) day in and day out, but my decades of experience in the industry shows that most positions aren't like that. I've not worked for google but I'm pretty sure it's also this way there. If you hire smart and competent people, it's OK even if they don't reverse a binary tree here or there.
Yes, exactly, since it's Google, they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate. Their policy might expect good candidate to get absorbed in Google one way or other, but they definitely want to filter a bad hire. However, in the recent years, I have seen this to be not working as expected. Still, I don't think it matters to Google as an big organization.
"filter out a bad hire"
Good teams have a balanced skill sets I've found. You need some uber-algorithm guys who can pass questions like this, but you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc.
Those types of engineers are what you consider a "bad hire", and won't be working at Google.
> you also need engineers who can fill in all the rest of the tasks that make up a production worthy system. I have not found that the uber-algorithm guys always pay attention to detail, put in good comments, think of all the edge cases, communicate well with outside teams, etc.
What's to keep a good engineer like you describe from buying a couple of books and spending a couple of months working on becoming the type of person that can do well on these questions?
4 replies →
"they can afford to have strong filters which will definitely filter out a bad hire, may be at the expense of rejecting a good candidate"
Can they, though? Obviously they believe they can, but are they correct?
Google's not going away any time soon, but they could easily slip a long way down from the top of the heap. There are signs that's started to happen.
I don't think they can be nearly as casual about losing top candidates as they seem to believe. In five or ten years we'll know, I suppose.
In the long run, this could affect Google, but, then again, if they change their hiring policies for better even after long 5 years, the best talent that might had been rejected before would still go to work at Google. So, still I don't think Google really consider it as an issue for them. Though, I have been burned once myself by them.
2 replies →
>>However, in the recent years, I have seen this to be not working as expected.
It never worked, what do you think happens when you start asking these kind of questions in interviews? People just go out and spend insane amount of time memorizing solutions to interview questions on the internet.
They continue the same after you have hired them. So their on the job productivity remains low. Please note, they need to prepare for their next job in the current one.
So the only thing this results is in people spending insane amount of time preparing interviews doing very little work in their day jobs.
I'd rather spend a year learning Python or Go or whatever, instead of spending a year practicing interview questions.
I never realized that some people put as much effort into practicing interview questions as I put into learning how to actually write software.
Does this mean technical interviews, even a structured interview, are completely useless? I.e., if you follow an interview script, eventually candidates find out about your script, and then it's useless.
This cracks me up a bit. I don't have anything against whiteboarding, even though I find it meaningless. With this you more often than not have false positives. And if Google had so many resumes they have to filter through, then why do they have all these recruiters literally spamming you to join? they have recruiting issues like many others tech companies, but the whiteboard is really a bad filter. I've seen very much capable,competent and experienced engineers being denied just because they didn't spend too much time learning cute algorithms or coding in the whiteboard, in front of interviewers sometimes more focused in showing off how smart they are.
"Google's perspective" shouldn't be a monolithic thing. Sure if you're sorting through candidates that were dropped on you then this process makes sense. But if you've discovered someone that by reputation and demonstrated code base has already excelled, and you're actively chasing them, then the interview process for that individual has to change. It's more about seeing if they'll thrive in your environment, and convincing them that you're an inevitable part of their career.
It's true that evaluating candidates is a hard problem, but you'd think that at the very least they would try to evaluate them based on skills that will actually be used in the job. Judging programmers based on their whiteboard skills is like judging a ballet dancer based on how eloquently he can describe his training routine.
Problem is if your system is actually capable measuring up to the claimed degree of accuracy, or just stabbing in the dark after a certain point.
IQ tests supposedly become inaccurate after 150/160, so you wouldn't really want to put too much emphasis on 180 vs 170 at those levels.
Well, my suggestion would be to use Page Rank to sift through the resumes. If you are respected by respected computer programmers then your "score" goes up. They could measure that by mentions in a blog, twitter "conversations", Github commits on the same project, etc.
"Hmmm... it looks like this guy really gets around. He's on the same project at GoodProgrammer A and GoodProgrammer B. He also tweets about functional programming and was mentioned in GoodProgrammer C's latest blog."