Comment by leeny
2 days ago
We did an experiment at interviewing.io a few months ago where we asked interviewees to try to cheat with AI, unbeknownst to their interviewers.
In parallel, we asked interviewers to use one of 3 question types: verbatim LeetCode questions, slightly modified LeetCode questions, and completely custom questions.
The full writeup is here: https://interviewing.io/blog/how-hard-is-it-to-cheat-with-ch...
TL;DR:
- Interviewers couldn't tell when candidates were cheating at all
- Both verbatim and slightly modified LeetCode questions were really easy to game with AI
- Custom questions were not gamable, on the other hand[1]
So, at least for now, my advice is that companies put more effort into coming up with questions that are unique to them. It's better for candidates because they get better signal about the work, it reduces the value asymmetry (companies have to put effort into their process instead of just grabbing questions from LeetCode etc), and it's better for employers (higher signal from the interview).
[1] This may change with the advent of better models
A couple years ago, we were using a take home coding assignment as a hiring signal. It was a small API based off something I'd built for an internal tool. It was self-contained and relatively easy to explain. The .md file was about two pages.
I recently fed it into ChatGPT and asked it to do the assignment. It did it perfectly -- I read the code in detail and couldn't find any issues.
So custom questions are off the table now, too. We'll be using a code review instead for the next round.