Comment by karaterobot
2 days ago
The best interview process I've ever been a part of involved pair programming with the person for a couple hours, after doing the tech screening having a phone call with a member of the team. You never failed to know within a few minutes whether the person could do the job, and be a good coworker. This process worked so well, it created the best team, most productive team I've worked on in 20+ years in the industry, despite that company's other dysfunctions.
The problem with it is the same curse that has rotted so much of software culture—the need for a scalable process with high throughput. "We need to run through hundreds of candidates per position, not a half dozen, are you crazy? It doesn't matter if the net result is better, it's the metrics along the way that matter!"
I dislike pair programming interviews - as they currently exist - because they usually feel like a time-crunched exam. You don't realistically have the freedom to actually think as you would in actual pair programming. i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview, but it's pretty realistic of real life work and entirely a non-problem. It's probably even good to test for at interview: how does a person work when they aren't working with an oracle that already knows the answer (ie: the interviewer)?
Pair programming with the person for a couple hours, maybe even on an actual feature, would probably work, assuming the candidate is compensated for their time. I can imagine it'd especially work for teams working on open source projects (Sentry, Zed, etc). Might not be as workable for companies whose work is entirely closed source.
Indeed, the other problem is what you mention: it doesn't scale to high throughput.
> i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview
In all pair programming interviews I have run (which I will admit have been only a few) I would fail myself as an interviewer if I was not able to guide the interviewee away from a dead end within 15 minutes.
If the candidate wasn't able to understand the hints I was giving them, or just kept driving forward, then they would fail.
Exactly! Who calls "researching how to build X, but then letting my pair-programming partner fall down a rabbit hole so I can feel superior" "pair programming".
1 reply →
> I would fail myself as an interviewer
You are not most interviewers, alas.
That's definitely up to the interviewer, in which a lot of discretion and trust has been placed. I think a lot of it also comes down to the culture of the company—whether they're cutthroat or supportive. As you get better people into the company, hopefully this improves over time. I know that when we did it, it was never about nailing it on the first try, it was literally about proving you knew how to program and were not an asshole. So, not the equivalent of reversing a binary tree on a whiteboard. The kinds of problems we worked on in the interviews weren't leetcode type problems, they were real tickets from our current project. Sometimes it was just doing stuff like making a new component or closing a bug, but those were the things we really did, so it felt like a better test.
Hiring doesn't scale, period; deal with it.
Exactly. People go like “the ideal way to interview is <whatever they themselves are the best at>”. Pair programming interviews suck and don’t scale, just like every other alternate way of hiring.
IMO interviewing is the biggest bottleneck and if interviewing was decoupled from hiring then it wouldn't be a problem. But this requires a Guild-like organization to manage interviewing/vetting and for companies to use said Guild for hiring. The companies could then do a single team culture meeting (if they wanted) before hiring.
6 replies →
For companies that successfully scale their team size, it literally did scale, right? I think you mean that hiring is very difficult to scale.
1 reply →
Exactly. Its almost like optimising for finding your best possible match for marriage. You don't go over a billion prospects, you choose from the ones locally available to you, as they come.
1 reply →
it isn't about scale. It is about core principle of the tech hiring - all the companies hire only the best. Not only it is impossible to scale, it is plain impossible. Even if all the companies hired only "above the average" it would still be a pretty tall order :)
13 replies →
> i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview
That’s an assumption. Perhaps following a dead end for a while, realizing it, pivoting, etc is a valuable, positive, signal?
I agree. But what I mean is: that's not how it's perceived in the current interview structure, which lasts maybe 45 minutes or so. Ultimately, going down a dead end means you'd now have 30 minutes to find the right solution and code it up. So the oracle (the interviewer) would probably help you realise sooner that it's a bad idea, so you don't waste your time. That's assuming they know the problem and solution well; if they don't, you'll just lose them and burn through your time.
In a 2 hour pair programming session on an 'unsolved' problem (like an open issue / minor bug / minor feature in a public repo), yes, it will likely not matter if you tried a bad idea, and would both be more realistic and a positive signal.
4 replies →
When I was doing interviews at my old job we did an hour of pair programming and we very intentionally designed the exercise such that
1) if you're qualified to do what we're hiring you for this stuff should be your bread and butter. We were doing restful web apps so the prompt might be "A restful API that takes in a string and returns that same string but backward." If you chase your tail on that for 15 minutes, you're not right for this job.
2) We weren't looking for a specific implementation. You want to make it a GET call with a query param that returns just the string? Neat. A POST with request and response DTOs? Also neat. We're not gonna ding you for doing it one way rather than the other but we're definitely gonna talk to you about why you made the choice that you did and maybe try to tease out another choice you could have made so that we can ask you to compare them. Again, the only wrong answer here is one you can't defend.
3) I'm not here to test your ability to memorize a particular language's syntax, your IDE's autocomplete or your ability to google. If you can articulate that you're trying to create a POST endpoint I'll give you that the correct annotation with Spring is @PostMapping or little things like that.
I do 1hr pair programming interviews for my company and you have to strike a balance between letting candidates think through the problem even when you think it won't work (to see their thought process and maybe be surprised at their approach working/see how quickly they can self-correct) and keeping them on track so that the interview still provides a good signal for candidates who are less familiar with that specific task/stack.
I'm also not actually testing for pair programming ability directly, moreso ability to complete practical tasks / work in a specific area, collaborate, and communicate. If you choose a problem that is general/expandable enough that good candidates for the position are unlikely to go down bad rabbit holes (eg for a senior fullstack role, create a minimal frontend and api server that talk to each other) it works just fine. Actually with these kinds of problems it's kind of good if your candidates end up "studying" them like with leetcode, because it means they are just learning how to do the things that they'll do on the job.
> maybe even on an actual feature
I don't think this would work unless the feature were entirely self-contained. If your workaround is to give the candidate an OSS project they need to study beforehand, I think that would bias candidates' performance in ways that aren't aligned with who you want to hire (eg how desperate are they for the role and how much time outside of work are they willing to put into your interview).
Another problem is it is difficult to compare candidates whose interviews involved working on completely different problems.
> if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview
Eh, if it's a reasonable bad end and you communicate through it, I wouldn't see it as a fail. Particularly if it's something I would have tried, too. (If it's something I wouldn't have thought of and it's a good idea, you're hired.)
I did a couple of rounds of this with my manager as the interviewer. Personally I really liked the process, and the feedback I got from the candidates was positive (but then again it always would be).
What worked well for me was that I made it very clear to my manager, a man who I trust, that I would not be able to provide him with a boolean pass/fail result. I couldn't provide him any objective measure of their ability or performance. What I could do was hang out with the canditates for an hour while we discussed some concepts I thought were important in my position. From that conversation I would be able to provide him a ranking along with a personal evaluation on whether I would personally like to work with the candidate.
I prepared some example problems that I worked for myself a bit. Then I went into the interviews with those problems and let them direct those same explorations of the problem. Some of them took me on detours I hadn't taken on my own. Some of them needed a little nudge at times. I never took specific notes, but just allowed my brain to get a natural impression of the person. I was there to get to know them, not administer an exam.
I feel like the whole experience worked super well. It felt casual, but also very telling. It was almost like a focused date. Afterwards I discussed my impression of the candidates with my manager to ensure the things I was weighing was somewhat commutable to what he desired.
All in all it was a very human process. It must have taken an enormous amount of trust from my manager to allow me the discretion to make a subjective judgment. I was extremely surprised at how clearly I was able to delineate the people, but also how that delineation shifted depending on which axis we evaluated. A simple pass/fail technical interview would have missed that image of a full person.
This is exactly how I interviewed people and what my replies were like. But I was one of many interviewers, so I would only take 15 minutes, which always annoyed my manager, but I never gave a bad "fine by me", so I still did interviews...
[dead]
I've (unfortunately) been interviewing the last two months and the main pattern that I've noticed is that a) big companies have terrible interview processes while b) small companies and startups are great at interviewing.
Big companies need to hire tons of people and interview even more so they need some sort of scalable process for it. An early stage startup can just ask you about your past projects and pair program with you for an hour.
> small companies and startups are great at interviewing
Small companies have the benefit of the pressure to fill a role to get work done, the lack of bureaucratic baggage to "protect" the company from bad hires, and generally don't have enough staff to suffer empire-building.
Somewhere along the line the question changes from "can this candidate do the job that other people in this office are already doing?" to "can this candidate do the job of this imaginary archetype I've invented with seemingly impossible qualities".
I hear this all the time, but I have yet to experience it. It may be because the small companies that I interview with are all startups, but I have yet to be able to get a call back from any other kind of small company. And the startups I do interview with have a full FAANG interview loops.
There seems to be a weird selection bias that if you're FAANG or FAANG adjacent these small companies aren't interested.
At a former gig we had a newly hired ex-facebook employee give notice within a month because she didn't like that dev setup had bugs that devs themselves had to fix. At fb they obviously can spend millions of dollars for a whole team that ensures that working dev env is always a button click away, a startup (even a scaleup) usually can't afford to. This is just one example out of many I can tell...
3 replies →
>”There seems to be a weird selection bias that if you're FAANG or FAANG adjacent these small companies aren't interested.”
Many smaller companies have noticed that former and wannabe FAANGers are looking for FAANG-type jobs, and are not good fits in their niche. Small companies often have more uncertainty, fewer clear objectives, less structure, and often lower pay. They’re not a good substitute for megacorps.
12 replies →
Yup. You can check out of FAANG anytime you like, but you can never leave.
Was path dependency for careers always this bad?
13 replies →
I've had a few good experiences with interviews at small companies and startups, so they do exist.
But I have also had really terrible experiences, similar to what you've mentioned. Sounds like you've just gotten unlucky and gotten the terrible ones.
Yeah, been there, done that; wannabe FAANGs are the worst.
11 replies →
What exactly does "scalable" mean here?
If a startup can spend 20 man-hours filling a single position, why can't a big company spend 1000 man-hours filling 50 positions?
In a small company, you can tell your buddy “just have a chat with the candidate and if you like them and you think they can do the job, hire them”.
If the person interviewing your candidates messes up, you’ll know soon enough. In a large company, the bad people will take over and your company is dying a slow death.
That approach doesn’t work on a large scale. Some interviewers are too nitpicky, elitist, others approve anyone who uses the same language as them for side projects. Some are racists, sexist, or have other kinds of biases. Some might have a crush on the candidate. Sometimes the interviewer thinks about their own task while they squeeze in an interview. In some countries, “undoing” a bad hire is hard, so they need to make sure that the candidate can work on any team (or at least on multiple teams reasonably well).
IMO for large companies it makes sense to standardize the interview process.
Also, in my opinion grinding leetcode is also a good personality check for FAANG hires: it shows the candidate can suck it up, study hundreds of hours, and do whatever they need to do to pass an arbitrary test, even if they themselves think it’s a broken process. The larger the company, the more this quality matters in candidates as they will need to deal with a lot of things they will probably not like.
8 replies →
> If a startup can spend 20 man-hours filling a single position, why can't a big company spend 1000 man-hours filling 50 positions?
Because big companies are run by bean counters and they also don't require the same kind of talent that is useful to startups. There's less competition for hyper-specialized seniors and middle of the pack generalists.
> If a startup can spend 20 man-hours filling a single position, why can't a big company spend 1000 man-hours filling 50 positions?
Huh. That’s actually a great question! I actually don’t know.
Was this an in-person pair programming session or remote via shared desktop?
Because the latter is garbage. You can't see the other person or read their body language. It also gives off "proctored test" vibes. Or worse than vibes - where you're expected to clear your room, swing your camera around, and never look away.
"We need to run through hundreds of candidates per position, not a half dozen"
But you don't! You only need to find the first person who is good enough to do the job. You do not need to find the best person.
You need to run through hundreds of candidates to find someone marginally qualified. I am not exaggerating.
Do we have different definitions of "marginally qualified"? Idk, I feel I'm a decent engineer - I can certainly do whatever leetcode medium they throw at me, as much as this counts of anything - and can actually code, but I still get maybe 1 callback per 50 applications.
Does "marginally qualified" mean "Ivy League Competitive Programmer PhD" or something?
8 replies →
>The best interview process I've ever been a part of involved pair programming with the person for a couple hours... You never failed to know within a few minutes whether the person could do the job
There is something funny about the "best interview process" taking "a couple hours" despite giving you the answer "within a few minutes". Seems like even the best process is a little broken.
Lightly ironic indeed! Though I’m not sure “broken” is exactly the word I’d choose.
I can only speak for myself, but I imagine myself as a candidate approaching a “couple of hours” project or relationship differently than I would a “few minutes” speed round. For that matter I can think of people I know professionally who I only know through their behavior “on stage” in structured and stylized meetings of a half hour or an hour—and I don’t feel like I have any sense at all of how they would be as day-to-day coworkers.
If we sat down to work together, you’d probably have a sense in the first few minutes of whether or not we were going to work out—but that would be contingent on us going into it with the attitude that we were working together as colleagues toward a goal.
That's mainly because there were multiple pairing sessions, and even if you knew the person was going to pass, there are still a couple more people who need to meet them, and a schedule to make sure they're available to do that. Plus due diligence, etc.
Nor am I saying it was a perfect system, just the best I've seen in terms of results.
This seems like a great idea, and similar to my old approach of “we pay you for 10-20 hours of work across a few days to see how you work with us”.
The downside we found was inconsistency. Candidate 1 and 2 get variable difficulty work. How do we decide who did better?
The biggest victims of these non-scalable process is people without a good network. As an intl PhD student, I am that person.
So now I have this weird dynamic: I get interview calls only from FAANG companies, the ones with the manpower to do your so called "cursed" scalable interviews. But the smaller companies or startups, ones who are a far better fit for my specialized kills, never call me. You need to either "know someone" or be from a big school, or there is zero chance.
Some of us find the prospect of pairing with an unknown person in an unknown environment, and against the clock, to be very stressful.
Anecdote:
I've been interviewing recently and got through to the last round (of five...) with an interesting company. I knew the interview involved pairing, but I didn't expect: two people sitting behind me while I sat on a wobbly stool at a standing desk, trying to use a sticky wired mouse and a non-UK keyboard, and a very bright monitor (I normally use dark mode). They had a screen recorder running, and a radio was on in the next room.
I totally bombed it. I suspect just the two people behind me would have been enough though.
I've had a great career and very positive feedback from every employer. I would do absolutely terrible in this kind of interview.
I would find trying to solve such problems with known people in known environments to be somewhat stressful too.
Pairing on something close to whatever real work they'd be doing, but familiar to the applicant is my favorite way to evaluate someone (e.g. choose a side project, pre agree adding a feature).
I don't care if someone uses modern tools (like AI assists), google, etc - e.g. "open book" - as that's the how they want to work. Evaluating their style of thinking / solving problems, comms, and output is the win.
Very few people doing this sort of interview (they tend to be our best, most empathetic developers) are likely to cut a multi-hour planned process short after a few minutes. It will eat at least an hour of their (very expensive & valuable) time.
Also how am I supposed to filter the 100's of AI-completed assessments? Who gets this opportunity?
We didn't do assessments (if by that you mean take home assignments). This was partly a solution to that, since nobody thought they were a good idea. If you mean the phone screen, I think that would be a problem, yep, but it wasn't an issue back in 2016. Having them pair would weed out cheaters, but we would have to figure out a way to weed them out during the screening, I agree.
We also did not require the employers doing the interview to be our most senior team members. They probably did it more often than most people, but often because they volunteered to do it. Anyone on the team would be part of the loop, which helped with scheduling. And, remember, we were working on actual tickets, so in a lot of cases it actually helped having the candidate there as a pairing partner.
For a little extra detail, the way we actually did it was to have 2-3 pairing sessions of up to 2 hours apiece. At the end of the day, all the team members who paired with the candidate had to give them the thumbs up.
Use another AI, of course!
Idk if I'm even being sarcastic here.
This. Interviewing for a sr dev position with a web app, backend stack is the bog standard java, spring, SQL abstracted away via JPA. We did a first screen, then the tech interview was two of their senior devs shoulder surfing me as I built a simple API. We chatted, I built, they asked questions, I defended my decisions (sometimes successfully, sometimes gracefully conceding defeat), they left knowing that I was who my resume said I was and the reminder that popped up in the middle of the interview to feed my sourdough starter showed them that I'm a culture fit.
I think you're onto something with that last paragraph but I want to try being a bit more generous with why things are the way they are. The question seems to be "When there are hundreds of applicants how do we give everyone a fair shake without hiring an entire team of devs who do nothing but interview?" From that perspective the intentionality is different and even sensible but the end product is likely to be the same. Even when someone is chasing a metric it's because someone else wants what's best and has decided that metric is a sensible way to make that happen. At the end of the day they really do want to hire the best candidate out of a pool whose size is extremely variable and that's challenging.
This is the exact time to use phrase “A people hire other A people, B people hire C people”
Additionally it’s rarely the hiring that makes a great team - it’s the long term commitment and investment in training.
I've been a proponent of pair programming since the early days of Agile, when it was still seen as part of extreme programming. Unfortunately, it’s not often employed in workplace settings.
With that said, would your perception of the interview remain positive if the outcome had been negative?
A common challenge across all interviews is a mismatch in personal dynamics, which can significantly impact the experience for both participants.
Consider a scenario where a senior developer, who prefers simplicity, is paired with a mid-level developer who is still captivated by complexity.
Or a "just start typing" person is with a "mull it over first" person. By the time I am typing code, I want to have 90% of it already completely worked out (at least till I type a "c.Lock()" and suddenly realize I hadn't considered thus and so synchronization issue.
> You never failed to know within a few minutes whether the person could do the job
Then why spend a couple hours?
I work at a company which has 11 engineers and competes with companies with 100s. The hiring process was a screening call with the CTO to not waste the prospective team's time, then a call with 2 of my prospective colleagues to gauge competence and cultural fit. Since then I have been involved in hiring most of the team I work with now. The CTO is one of the most competent engineers I have ever met and he designed this process. He also has very high EQ. One of the points I sell to prospective hires is him as a person to work with, as well as our team. He has also flatly denied people I suggested before and that's fine.
I have been here 5 years now and I'm working with the most competent team I have ever worked with. My take away from this is that hiring doesn't need to be commoditised and scale, it just needs to find good people and give them an opportunity to show you that you do or don't want to work with them.
Similarly, in general the best interviews I've ever been part of (either giving or being) turn into discussions where people's experience, opinions, and stories get aired (going both ways). You eventually get a good sense of each other and things get more relaxed when you both realize that you know what you're talking about (this is harder for Jr roles, though).
Being peppered with questions very rarely gives any insight.
For junior roles, you want to interview for intelligence and shall we say an interest in learning rather than specific skills.
Even for senior roles, that's what I want to interview for, although it is true at times a business case can be made for someone that is good at some specific complex skill and doesn't need to listen to other people to do ok work.
> person for a couple hours,
>You never failed to know within a few minutes whether the person could do the job
Did a misunderstood something or your best interview process is to multiple hours from someone when you've decided within minutes?
Pair programming on what problem though? I dont think many companies would want an outsider to work on their codebase.
I used to love getting to know the interviewer and doing things like that but IMO the market has shifted fundamentally on both ends for this to be effective anymore for most SaaS roles. This is anecdotal for US/Canada tech market over the past 10 years so YMMV.
Developers Side: Since developers don't have job security anymore (at least for those who work on common languages like Go, Python, Java and Typescript) they are better off learning and keeping in touch with leetcode and system design questions, looking for new opportunities and interviewing in "batch mode" when looking for a job. The idea is to clear as many interviews as possible using the same concepts, get in and make money asap before you get laid off. No incentive for collaboration or for fulfilling but esoteric stuff like Haskell and Scala. Career security > Job security.
Companies Side: On the other end software companies have less trust in developers staying long term so they want to make the interview process as quick and risk free as possible. In essence they are betting that by perusing 100s of resumes and hiring someone who seemingly knows CS concepts they can get some value out of them before they leave. Standardized tests/vetting > team fit.
TLDR; The art is gone from this job, its become akin to management consulting or investment banking. Quality and UX seems to be regressing across the board as a result.
>its become akin to management consulting or investment banking
Not sure how those are similar.
I meant the “grind”, short term profit mentality of SWE market has become similar to professionals in those fields, not that any of these fields are similar.
This is how my team hires and it’s incredible.
I think what makes it work is that our code pair is pretty low stakes. I was told that I didn’t have to finish the problem and I was free to use whatever tools or language I needed. They just wanted to see how I work and collaborate.
It's a super interesting approach, and if you put a strong filter before it (e.g. intense non-BS Q&A), the whole thing could be high-throughput.
Does your company operate the same way? I.e. is most, or at least a large chunk of engineering done as pair-programming?
Thats what we did, pair program on some real production code and tickets. This way the person could get a feel about what they potentially were walking into and you get a good idea of how they think and approach problems.
Beautifully articulated truth