Comment by enraged_camel
16 hours ago
The problem with work-sample testing (which is commonly administered as a take-home problem for the developer candidate to solve) is two-fold:
a) it discriminates against people who cannot spare 4+ hours of focused time on evenings/weekends to work on the problem. People with multiple jobs, single parents, etc.
b) in the age of AI it is no longer a reliable measure of someone's skill, for obvious reasons
Unlike Yegge, I haven't worked at FAANG, but the companies I have worked at all followed the same hiring practices and suffered from the same problems as he describes.
Provisional employment (or, if that's not possible, then well-paid internships) solve all of those issues. The candidate gets 3-6 months of stable employment, you as the employer get a large number of work-sample tests, and you can see how they use AI and how much.
And coding interviews often bias against people who can’t spend days of time grinding on hackerrank or whatever to prepare for the interview. Provisional employment biases against people who currently have jobs.
There is no perfect interview process, so it’s important companies think about what they’re biasing for or against. A work sample gives a more realistic picture of someone’s ability than a coding interview, but less than provisional employment. But, it’s much easier for people to commit to finding the time for it than to spending 3-6 months in limbo.
Regarding AI and work samples, it seems like this problem is largely solved by having an in-depth conversation with the candidate about the assignment, no? They should be able to explain why they made the decisions they did, pros and cons of their approach, etc. If they can do that, and their solution is good, does it matter if it was LLM-assisted?
>The candidate gets 3-6 months of stable employment
To me, "stable" implies I don't even worry about having a full-time job for the next 3-5 years. Anything less isn't stable, unless this would be my second job immediately after the McDonald's burgerflipper one where I was scheduled for random 4 hour shifts seemingly designed to maximize personal inconvenience.
If I knew that my job would only last 6 months, I would have to immediately begin conducting a job search and prepping for it. I put my odds at finding another job before the 6 months is up at less than 60%. Even that number seems naively optimistic now that I've typed it out. Offering me 3-6 months of employment only sounds like a deal if I'm currently unemployed and rent was due yesterday.
I think a very real problem is that these take home problems are often way more than 4 hours. And to that they often add the traditional 4-6 hour interview loop.
Provisional employment doesn’t work for most cases, though. It might attract people who have no job and it might attract people who have so much saved that they are okay with potentially being let go after 90 days. But I imagine the vast majority of the potential employment pool are not willing to quit their current job to accept a “maybe” job from another company.
Adding take-home problems to a traditional 4-6 hour interview loop is odious.
But the "way more than 4 hours" thing smuggles in a premise: that every candidate should be able to finish the challenge in the allotted time. But candidates with greater aptitude or conversance with the problem domain will complete work sample tests faster than candidates without, and selecting for those candidates is the point of hiring qualification.
It depends on the details of the work sample test.
If I ask you to write me a python function to convert OSGB easting/northing into WGS84 longitude/latitude the task has a very clearly defined scope. If you knock it out in a quarter of the allotted 4 hours, you've saved time. You can't use the remaining time to go further and demonstrate your mastery.
On the other hand if I ask you to write me a website for organising photos, there's no such thing as 'done' - no matter how good you are, after 4 hours you'll still be able to think of ways to make it faster, more beautiful, more featureful, more scalable, cheaper to operate, etc
Obviously, as a hiring manager I'll notice if you've spent 40 hours on the 4 hour task - but if you've spent 6 hours maybe I just think you're a fast worker with relevant experience and sharp tools. And my sense of how far you can get is calibrated by other prospective hires; if lots of people are spending 6 hours and claiming to have spent 4, my expectations will naturally be high.
17 replies →
This is theoretically true but it’s also rife with misaligned priorities. The people putting together these take home assignments have little incentive to ensure that they can be completed by a competent engineer in the allotted time. The engineers completing these assignments are definitely incentivized to underreport how long they spent on the assignment.
With AI coding this is also largely useless. These “build this thing in 4 hours” assignments come with a literal prepared prompt so that they can be churned out in 10 minutes.
4 replies →
A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.
The AI thing is an interesting problem, but a solvable one. We continue to hire resume-blind.
The question is whether companies would tolerate their own process if their employees did that recruiting process at a different company. They obviously do to some extent via plausible deniability; I have a 1 hour "doctors" appointment this afternoon, or I'm taking leave on Monday. Using it as cover to attend an interview.
Would this company permit an employee taking 3 months unpaid leave to provisional hire somewhere else and have free choice whether they stay or go at the end of it.
I feel like you could get around the AI bit by asking about components and what they do, rationale for decisions, etc. If someone can't speak to it, it should be a clear tell.
We hire entirely based on work sample testing, and there's a lot of stuff you can do to make it work in with AI-equipped candidates; I'm not prepared to write it up at the moment, but you start by recognizing that everybody is going to be using AI and designing the tests accordingly, and by relying on unassisted interactive challenges as a component of the process.
As long as you are talking to them face to face; over the phone they will use AI with speech recognition and parrot its response, erasing all signal. Then the interview becomes all about AI detection.
1 reply →
>> A standard interview loop kills an entire work day, and is preceded by phone interviews that eat several hours. Properly budgeted work samples are strictly better from the candidate's time perspective, not to mention that you can do them from your couch rather than under flourescent lights in a confeence room.
Yes, standard interview loops also discriminate, and the more time they take, the more discriminatory they are. Any on-site requirements compound the issues.
Like Yegge says: provisional employment/internships solve all of these issues. You get the best of all worlds: stable employment for the candidate where they get paid a regular wage and aren't under a stressful interview setting, and lots and lots of work samples for you, the employer. It's not perfect. For example, it can be difficult to entrust the provisional employee/intern with anything impactful if you don't know whether they'll be employed at the end. But it is significantly better than the alternatives in most contexts.
Provisional employment does not work. It requires candidates to leave their jobs before they know whether they have a secure job with your firm. I concede that provisional hires are higher-signal than work sample testing (or rather: that they're the platonic ideal of work sample testing), but the entire problem of hiring qualification is to make decisions in the context of a candidate doing a job search.
2 replies →
Provisional employment only makes sense if you work remotely and have full benefits (esp. healthcare). Moving is costly.
1 reply →
We do all our work samples in person at a our offices as part of the in person interviews. Takes 2-3 hours, never been a problem so far.
If you are going to take a day off to do 3-4 in person interviews at a company then this slots in well.