Comment by zarzavat
17 days ago
In the UK the driving test requires a portion of driving using a satnav, the idea being that drivers are going to use satnavs so it's important to test that they know how how to use them safely.
The same goes for using Claude in a programming interview. If the environment of interview is not representative of how people actually work then the interview needs to be changed.
In the Before Times we used to do programming interviews with “you can use Google and stack overflow” for precisely this reason. We weren’t testing for encyclopaedic knowledge - we were testing to see if the candidate could solve a problem.
But the hard part is designing the problem so that it exercises skill.
We don't solve LeetCode for a living yet it is asked in interviews anyway, so nah, we don't have to use AI in interviews.
You’ve just written the exact reason LeeteCode is widely mocked as an interview technique. They are not representative of most real world software, and engineers that train to solve them give a false impression of their ability to solve most other problems.
I’ve interviewed hundreds of engineers for software and hardware roles. A good coding test is based on self-contained problems that the team actually encountered while developing our product. Boil the problem down to its core, create a realistic setup that reflects the information the team had when they encountered the challenge, and then ask the candidate to think it through. It doesn’t matter if they only write notes or pseudo code, and it doesn’t matter if they reach the wrong conclusion. What it’s testing for is the thought process. The fact the candidate has to ask the interviewer questions as though the interviewer is effectively the IDE, is great! The interviewer experiences the engineer’s thought process first-hand. And the interviewer can nudge the candidate in the correct direction by communicating answers that aren’t just typical IDE error messages.
To validate these kinds of questions in advance, I’d often run them on existing team members that hadn’t already been exposed to the real challenge the problem was based on.
Leetcode's utility is not in showing you can solve real-world problems. It's used as a baseline to estimate how smart you are. Every shop prides itself on hiring smart people, and some only want the best of the best—your MIT and Stanford grads, etc. A smarter engineering workforce can not only solve the problems you have, they're better positioned to spot and avoid problems you haven't anticipated yet. Anyways, IQ testing as a condition of employment can open you up to legal liability, as IQ tests are horribly racist. Leetcode is a way around that.
1 reply →