Comment by csallen
3 days ago
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
If anthropic is making as many PRs as stated in articles, then rolling back is unlikely an option. More likely it'd be a new commit to fix it, and with a feature flag to preserve the "wrong" behavior (even if wrong, since people will probably depend on that function since its original behaviour).
Those sorts of changes are really rough to manage. Maybe still manageable by LLMs but we'll find out.
> can roll back changes and investigate them
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
> Just like people are more careful when carrying a baby than when carrying a pillow
lol, I assume you’re not a parent then. Or at best a new parent.
3 replies →
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
Is the Hugging Face incident from July really a problem? Or Australian healthcare portal breach? Who let their guard down unable to detect or roll back the change for days on end?
As I said at the end of my comment, unless you're an AI researcher pushing the bound of what LLMs can do. I don't think most of us are running ten million dollar AI swarms with unreleased models that are escaping their sandboxes. That's research, not software engineering, and this discussion is about whether or not software engineers can reasonably expect to understand their LLM generated code.
Also, for what it's worth, LLM research is hardly the first scientific research to have accidents with negative external consequences.
Frontier labs are not the first to socialize externalities and privatize the gains. Not even the first to shift the responsibility on to some non-entity (agents, corporations). I think you have a very reductionist view of software. Software is used to design the buildings you inhabit, bridges you cross, vehicles you drive, medical devices you wear...
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
"their code" == code they actually write, that originates with them, "their system" == anything, from gentoo to something off-the-shelf
So the real question would be, how many people designing / implementing the boot process of any given system don't understand what what's going on.