Comment by enraged_camel
16 hours ago
The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold:
a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc.
b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons
Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes.
Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.
I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop.
Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.
Adding take-home problems to a traditional 4-6 hour interview loop is odious.
But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
It depends on the details of the work sample test.
If I ask you to write me a python function to convert OSGB easting/northing into WGS84 longitude/latitude the task has a very clearly defined scope. If you knock it out in a quarter of the allotted 4 hours, you've saved time. You can't use the remaining time to go further and demonstrate your mastery.
On the other hand if I ask you to write me a website for organising photos, there's no such thing as 'done' - no matter how good you are, after 4 hours you'll still be able to think of ways to make it faster, more beautiful, more featureful, more scalable, cheaper to operate, etc
Obviously, as a hiring manager I'll notice if you've spent 40 hours on the 4 hour task - but if you've spent 6 hours maybe I just think you're a fast worker with relevant experience and sharp tools. And my sense of how far you can get is calibrated by other prospective hires; if lots of people are spending 6 hours and claiming to have spent 4, my expectations will naturally be high.
17 replies →
This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment.
With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
4 replies →
A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.
The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.
The question is whether companies would tolerate their own process if their employees did that recruiting process at a different company. They obviously do to some extent via plausible deniability; I have a 1 hour "doctors" appointment this afternoon, or I'm taking leave on Monday. Using it as cover to attend an interview.
Would this company permit an employee taking 3 months unpaid leave to provisional hire somewhere else and have free choice whether they stay or go at the end of it.
I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
1 reply →
>> A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.
Yes, standard interview loops also discriminate, and the more time they take, the more discriminatory they are. Any on-site requirements compound the issues.
Like Yegge says: provisional employment/internships solve all of these issues. You get the best of all worlds: stable employment for the candidate where they get paid a regular wage and aren't under a stressful interview setting, and lots and lots of work samples for you, the employer. It's not perfect. For example, it can be difficult to entrust the provisional employee/intern with anything impactful if you don't know whether they'll be employed at the end. But it is significantly better than the alternatives in most contexts.
Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. I concede that provisional hires are higher-signal than work sample testing (or rather: that they're the platonic ideal of work sample testing), but the entire problem of hiring qualification is to make decisions in the context of a candidate doing a job search.
2 replies →
Provisional employment only makes sense if you work remotely and have full benefits (esp. healthcare). Moving is costly.
1 reply →
We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far.
If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.