Comment by baalimago
18 days ago
Well, the coding agent is pretty much a junior dev at the moment. The seniors are teaching it. Give it a 100k PRs with senior developer feedback and it'll improve just like you'd anticipate a junior would. There is no way that FANG aren't using the comments by the seniors as training data for their next version.
It's a long-term play to have pricey senior developers argue with an llm
> using the comments by the seniors as training data for their next version
Yeah, I'm sure 100k comments with "Copilot, please look into this" and "The test cases are still failing" will massively improve these models.
Some of that seems somewhat strategic. With a junior you might do the same if you’re time pressured, or you might sidebar them in real life or they may come to you and you give more helpful advice.
Any senior dev at these organizations should know to some degree how LLMs work and in my opinion would to some degree, as a self protection mechanism, default to ambiguous vague comments like this. Some of the mentality is “if I have to look at it and solve it why don’t I go ahead and do it anyways vs having you do it” effort choices they’d do regardless of what is producing the PR. I think other parts of it is “why would I train my replacement, there’s no advantage for me here.”
Sidebar? With a junior developer making these mistakes over and over again, they wouldn't even make it past the probationary period in their employment contract.
2 replies →
These things don't learn after training. There is no teaching going on here, and the arguments probably don't make for good training data without more refinement. That's why junior devs are still better than LLMs IMO, they do learn.
This is a performative waste of time
A junior dev is (most often) a bright human being, with not much coding experience yet. They can certainly execute instructions and solve novel problems on their own, and they most certainly don't need 100k PRs to pick up new skills.
Equating LLMs to humans is pretty damn.. stupid. It's not even close (otherwise how come all the litany of office jobs that require far less reasoning than software development are not replaced?).
A junior dev may also swap jobs, require vacation days, perks and can't be scaled up at a the click of a button. There are no such issues with an agent. So, if I were a FANG higher-up, I'd invest quite a bit into training LLM-agents who make pesky humans redundant.
Doing so has low risk, the senior devs may perhaps get fed up and quit, and the company might be a laughing stock on public PRs. But the potential value for is huge.
It's probably easier to make the higher up redundant than to actually achieve high speed and predictable outcomes that satisfy real business needs and integrations in a cost effective way.
I mean, a Furby could respond to you all day, each hour, but that doesn't make them any more useful..
Not saying that LLMs are useless, but that's a false equivalency. Sure, my auto complete is also working 0-24, but I would rather visit my actual doctor who is only available in a very limited time frame.
> Give it a 100k PRs with senior developer feedback
Don't you think it has already been trained with, I don't know, maybe millions of PRs?
at the very least, a junior shouldn't be adding new tests that fail. Will an LLM be able learn the social shame associated with that sort of lazy attitude? I imagine its fidelity isn't detailed enough to differentiate such a social failure from a request to improve a comment. Rather, it will propagate based on some coarse grained measures of success with high volume instead.
I’m curious why you think it hasn’t already been trained on 100ks or millions of PRs and their comments/feedback.
@Grok is this true?