Comment by gwd
1 day ago
I've been on the other side of this interaction, and it's just as frustrating -- we do a phone interview a student who's been doing random stuff with the project for a while; and some combination of stress, the phone, not being in his native language, and he's totally not performing, in a way that seems almost certain to be situational and temporary. We were all open to trying some other format that would help him perform better, but he decided to cut his losses and apply elsewhere.
But unfortunately, it seems pretty likely to me that non-live coding interviews will measure the ability to query LLMs. If it's a choice between filtering out people who freeze up at interviews, or letting in scammers who have no ability whatsoever, I'm afraid the first is still the best option.
> If it's a choice between filtering out people who freeze up at interviews, or letting in scammers who have no ability whatsoever, I'm afraid the first is still the best option.
So far, my opinion is: you still can weed out scammers by running your questions against an LLM in advance. There are still easy questions (not even involving coding, just reasoning about the problem and identifying edge cases) where LLMs fail spectacularly.
If you think they're good, why not do a paid day of work then? They'll probably be querying LLMs all day after you hire them anyway.
I love this idea in theory, but in practice this would just mean coming up with some throwaway project and then paying them a full day to build on it. I've never had a job where there wasn't at least a few days of onboarding (often times a few weeks) before somebody could productively contribute to our applications. If it's a brand new project then sure, but how often does that happen? And when it does, do you want your completely unknown candidate doing it or one of your senior engineers who understands the big picture, how it will fit in, your company culture/values/etc?
My experience is LLMs can suggest some pretty good multi-day throwaway projects. I'd want to know can the candidate manage multiple agents, and effectively communicate/collaborate with humans.
Even something simple, like take 1 hr then explain to us this new repository in detail.
It doesn't need to be a day then hire. It can be done in stages.
I think this is impractical without at least some filter first. Some jobs get thousands of applications and some of those are fully fraud. I'm all for a paid work stint but where do you draw the line, how do you evaluate it fairly, and how do you make that possible and attractive for people already working?
Also: as someone who interviews and handles hiring, the biggest bottleneck for me is my time.
In enterprise sure, I had whole weeks where not much was going on, but in tech or in startups? An hour is already precious.
> If you think they're good, why not do a paid day of work then?
not many people could take the time away from their current job, so you'd be automatically filtering out people who are currently employed (which, i would imagine, is a signal that they're already sufficiently good).
So only those who could take the time can come to this style of interview, and it introduces a systemic bias.
I never quite got this argument. If someone isn't willing to make time to work on a PAID project, it means they don't want the job bad enough. Take a day of vacation geez...
1 reply →
If you know you're good, why don't you offer to pay to do a day of work for them? The potential upside seems massive if you actually get the job. You would also be able to offset the costs of borrowing another engineer's time to get up to speed somewhat.
That's a bad idea in general. A paid day of work may be legally impossible for some candidates who are currently employed and have signed an agreement with their current employer regarding IP rights and/or conflict of interest.
The way I've seen it implemented is that companies just wire it via Paypal and don't think too hard about it.
Wait, I meant startups. So compliance? Probably not a thing.
1 reply →
This would be at least partly my solution. Issue as I see it is, hiring someone is expensive largely because you're committing to giving them a prolonged position thats difficult to terminate.
I think contract hires need to become more of a norm.
Contract-to-hire positions are generally only attractive to candidates who aren't currently employed. For those who are, it's too big a risk.
Managing all those 1099s come tax day as an IC looking for a job sounds nightmarish, let alone from the company's perspective.
> They'll probably be querying LLMs all day after you hire them anyway.
At least they hey will know when LLMs make mistakes.
Companies constantly talk about how expensive it is to hire people because your most valuable employees have to spend time prepping for interviews, reviewing homework, discussing results etc.
In the era of full remote work I don't understand why you don't just have a small greenfield project that you can quickly onboard prospective employees with. Have a brief personality / resume griller screening, then hire them for a week and toss them at a few bugs or issues for this project with a semi-public slack. Then see what they output and decide at the end.
That would likely be less expensive then all the other nine layers of interviewing hell they maintain and get better results.
I realize my field is kinda special (industrial automation) in that it's really a mix of programming, IT, and electrical engineering, but I can usually tell if someone is knowledgeable with just a few minutes of conversation. You can also weed out the bullshitters - the people that present guesses as fact - who in my experience are much more of a problem.
I know of no test that will tell you if they're a good fit for your company, which is arguably more important than their skill level.
I guess I don't really understand what the coding tests are supposed to achieve, unless you're letting your managers interview people without a senior programmer to help. If that's the case, you may as well just pick randomly.
> I know of no test that will tell you if they're a good fit for your company, which is arguably more important than their skill level.
I'm beginning to see it the other way around though. My manager has to "get me". I've only met one so far and he understood who I was well enough during our first interview.
He was like "ah I see, so you have lots of different skills and you want to go into lots of different directions. Awesome! I think we can do amazing projects together." This was coupled with "during lunch we like to talk about (geo) politics, some science facts, random stuff and self-improvement, those type of things." (note: Dutch company)
Ultimately I noticed he must have a fairly high openness to experience. I think all managers that I am managed by should have that.
Having a manager that knows how to properly employ you is important, sure. I was talking more about how you work with your teammates to get things done.
There's a place for lone wolves at some companies, but in my line of business you have to be able to work with engineers, project managers, electricians, as well as other integrators. Our projects are definitely team efforts. I make an effort to assign tasks based on people's strengths, but if you're hard for my team to work with, you're useless to me and I'll eventually be in the boss' office asking for you to be replaced.
Don't university courses at some point entail open book, internet access enabled exams? And that's good enough for passing or failing a whole semester?
> will measure the ability to query LLMs.
Hell, most places are requiring their developers to query LLMs now anyway.
If your interview can be passed by using an LLM and the job you are hiring for can’t be done solely with an LLM, by definition your interview isn’t testing for the skills you need for the job.