Comment by wyager
3 days ago
Leetcode with no prep is a pretty decent coding skill test
The problem is that it is too amenable to prep
You can move your score like 2stddev with practice, which makes the test almost useless in many cases
On good tests, your score doesn't change much with practice, so the system is less vulnerable to Goodharting and people don't waste/spend a bunch of time gaming it
I think LC is used mostly as a metric of how much tolerance you have for BS and unpaid work: If you are willing to put unpaid time to prepare for something with realistically zero relevance with the day-to-day duties of the position, then you are ripe enough to be squeezed out.
Cynical, but correct. I've long maintained that these trials, much like those we encounter in the school system, are only partially meant to test aptitude. Perhaps more importantly, they measure submissive compliance.
It selects for age and childlessness.
And experience selects for age as well, doesn't make it a bad signal.
> On good tests, your score doesn't change much with practice, so the system is less vulnerable to Goodharting and people don't waste/spend a bunch of time gaming it
This framing of the problem is deeply troubling to me. A good test is one that evaluates candidates on the tasks that they will do at the workplace and preferably connects those tasks to positive business outcomes.
If a candidate's performance improves with practice, then so what? The only thing we should care about is that the interview performance reflects well on how the candidate will do within the company.
Skill is not a univariate quantity that doesn't change with time. Also it's susceptible to other confounding variables which negatively impact performance. It doesn't matter if you hire the smartest devs. If the social environment and quality of management is poor, then the work performance will be poor as well.
> A good test is one that evaluates candidates on the tasks that they will do at the workplace
Systematizing this is not feasible. The next best thing (in terms of predictive power for future job success) is direct IQ tests, which are illegal in the US. Next best thing after that are IQ proxies like coding puzzle ability.
> If a candidate's performance improves with practice, then so what?
It means the test isn't measuring anything useful. The extremely broad spectrum skills that benefit a software/eng role aren't something you can "practice".
> The only thing we should care about is that the interview performance reflects well on how the candidate will do within the company.
Agreed, which any Goodhartable test will never do.
Direct IQ tests are not illegal in the US. Several very large companies deliver them to candidates; the most prominent company that administers general cognitive exams for employment purposes has a logo crawl on its front page with names your parents would recognize. If this persistent meme about them being forbidden was real, employment lawyers would be making bank off them. The real reason most companies don't do general IQ testing for candidates is that it's not an effective screen for aptitude.