Comment by perlgeek
2 days ago
Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?
There's very likely a real answer to that question, and that answer should shape the way that engineer should be assessed and hired.
For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.
It seems to me the heart of the problem is that companies aren't very clear about what value the engineers add, and so they have trouble deciding whether a candidate could provide that value.
The even bigger challenge is that hiring experts in any domain requires domain knowledge, but hiring has been shifted to HR. They aren't experts in anything, and for some years they made do with formulaic approaches, but that doesn't cut it anymore. So now if your group wants to get it done, and done well, you have to get involved yourself, and it's a lot of work on top of your regular tasks. Maybe more work because HR is deeply involved.
>hiring has been shifted to HR
Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum.
My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)
Personal networks only disadvantage those who have none.
28 replies →
I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.
Yup, I have seen some really poor decisions as a result of this. I'm also curious - what will be the effect of AI assistance during behavior interviews, etc.
HR are experts in HR, which is to say they have a broader view of the institutional needs and legal requirements of hiring staffing than you do. It's always annoying when that clashes with your vision, but dismissing their entire domain is unlikely to help you avoid running into that dynamic again and again
> hiring has been shifted to HR
Not everywhere. At my company, HR owns the process but we -- the hiring tech team -- own the content of interviews and the outcomes.
I've never seen hiring completely in the domain of HR. HR filters incoming candidates and checks for culture fit etc, but technical competency is checked by engineers/ML folks. I can't imagine an HR person checking if someone understands neural networks.
HR involvement is unavoidable at big companies; and basics like "years of experience for payband" can cause issues. They fundamentally do not understand the job, but somehow have to ensure its not a biased hiring process.
1 reply →
> Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?
Interview coding questions aren't like the day-to-day job, because of the nature of an interview.
In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.
It also has to be a problem that's solvable within about 40 minutes.
The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me.
And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind.
And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.
> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.
If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan.
You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.
> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.
Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the depths of their knowledge. I don't get why we are over-engineering interviews. If I have 10+ years of experience with some proof through chatting that I am, in fact, a professional software engineer, isn't that enough?
Have you ever hired?
No, it's not enough. There are people out there who can talk great talk, and have great resume, but cannot do their actual job for some reason. Maybe they cannot read the code, maybe they cannot write the code, maybe they can write the code but not in the manner that keeps the rest of codebase working... I've had people like that on my team, it was miserable for all of us.
It is essential to see candidate actually write and debug code. It would be even better if we could see how the candidate deals with existing huge codebase, but sadly this kind of thing can't be easily done in a quick interview, and good candidates don't want trial periods.
>Interview coding questions aren't like the day-to-day job, because of the nature of an interview.
You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.
That’s silly. It’s like saying that a car can do a 100m dash faster than a human so it’s not a useful test for selecting players for an NFL team.
One of the best interviews I've encountered as a candidate wasn't exactly a pair programming session but it was similar. The interviewer pulled up a webpage of theirs and showed me a problem with it, and then asked how I would approach fixing it. We worked our way through many parts of their stack and while it was me driving most of the way we ended up having a number of interesting conversations that cropped up organically at various points. It was scheduled for an hour and the time actually flew by.
I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.
Problem is, company A doesn't need an engineer to solve those interview questions but real problems.
“Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.
On that, these unreal questions/problems are decent proxies for general knowledge for humans, but not for AI. Humans don't have encyclopedic knowledge, so questions on a topic can do a decent job of indicating a person has the broader depth of knowledge in that topic and could bring that to bear in a job. An AI can answer all the questions but can't bring that to bear in a job.
WE saw this last year with all the "AI can now pass the bar exam" articles, but that doesn't lead to them being able to do anything approaching practicing law, because AI failure modes are not the same as humans and can't be tested the same way.
Really? How short are your interviews, and how big are these Real Problems such that you can't get a sense of how your candidate would start to tackle them?
7 replies →
This is the answer.
Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.
Let's go one step further: LLMs can't solve anything, but most interview questions are covered so much online that they'll parrot a passable answer.
1 reply →
Tech interviews in general need to be overhauled, and if they were it’d be less likely that AI would be as helpful in the process to begin with (at least for LLMs in their current state).
Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on.
AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.
I want to hire people who can be given some problem and will go off and work on it and come to me with questions when specs are unclear or there's some weird thing that cropped up. AI is 100% not that. You have to watch it like a 15 year old driver.
A company wants to hire someone to perform tasks X, Y and Z. It's difficult to cleanly evaluate someone's ability to do these tasks in a short amount of time, so they do their best to construct a task A which is easy to test, and such that most people who can do A can also do X, Y and Z.
Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult.
This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.
It's because coding interview questions aren't so much assessing job skills as much as they are thinly veiled IQ tests.
I think if it was socially acceptable they'd just do the latter.
Plenty of companies administer IQ tests. The reason everyone doesn't is that it doesn't work well.
Nothing works well but IQ tests predict job performance better than anything else.
13 replies →
A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.
McK's case interview is just as game-able as HackerRank style interviews. There are entire consulting clubs at many colleges that teach this exact interview style. It's true that it's harder (but not impossible) to use AI to help, but calling it an IQ-like test is true only as much as any other technical interview.
That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .
3 replies →
And they're losing all but the worst candidates because of it, which explains a lot.
This is a great point. Though what if the answer is that the company can hire that AI to solve a significant fraction of its actual problems? People who do the assessments and decide what features should look like are often called managers (product, engineering, etc.).
For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management.
If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.
> For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.
So a Product Manager?
Maybe.
Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools.
Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work.
But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.
In most companies every engineer above a junior level is expected to pass features and bugfixes through their common sense filter and provide feedback. Product managers and designers aren't infallible and sometimes lack knowledge about the system or product that an engineer might have.
You can't just take requirements and churn out code without a critical eye at what you're doing.
[dead]