← Back to context

Comment by OtherShrezzing

2 months ago

We’ve seen this in a few open source repos we voluntarily manage security on. They’re not massive repos, but big enough they get attention from security researchers.

Most of the real low hanging fruit was picked up by humans years ago. When doing automated scanning, the majority of stuff is overly-verbose nonsense which takes hours of expert human labour to understand, test, and discard.

Reading through a Claude generated false positive is absolutely excruciating, because it is absolutely determined that what it’s found is justified. Often you’ll receive very long accompanying “proof of concept” code which demonstrates absolutely wild scenarios. It’s especially frustrating when you’re volunteering your time for a project, and a well-meaning contributor submits the report without the technical nous to understand why you’re rejecting it.

Before:

There is a vulnerability in library X when you call Y with these specially crafted parameters; as seen in the attached logs it can overflow buffer Z and clobber memory potentially leading to an RCE.

After:

Honest take: the load-bearing constraint violation is real. The log documenting the exploit gate is the ledger which weaves the story.

Right now, all top tier LLMs are as eager, bright 20ish year old interns.

Very gung ho, full of energy, loads of book learning, no real world experience or understanding of why things are as they are.

Leave them to their own devices at your peril. Trust nothing they do.

Yet directly guide them, monitor everything they do, some value emerges.

  • > Right now, all top tier LLMs are as eager, bright 20ish year old interns. > Yet directly guide them, monitor everything they do, some value emerges.

    That's simply not true. I've had some very talented interns, and they are leagues ahead of what the LLMs can do. Not that it matters though, because the point of having interns wasn't to have them produce value. What made the investment worth it was that 12 months down the line I would have a competent colleague that I could have an interesting conversation with. A human person that could challenge some of my blind spots. A person that could take responsibility of something. Maybe not my most important work, but some of it. You don't get ANY of that from the LLM.

  • I wish people stopped equating LLM to interns or junior engineers. They are tools, and as good they may be at some specific things people do, they also really suck at many more which we wouldn’t find acceptable in humans.

    • Indeed they are tools. But if people/companies treat them as tools that can take a problem a human used to solve, and make them solve it from start to end, then the comparison begins to be necessary.

    • It's merely a way to frame things, in terms of experience and trust. And it highlights how an LLM can code very well, but not truly understand the ramifications of that code.

      1 reply →

  • I sure hope they stop training them to be spaghetti throwers, please stop and ask questions when there is ambiguity

    • Well let's see. You sell to management. Does management buy:

      1) nuanced tools that talk back, question assumptions, take over decisions, ... oh and expose just how much management knows about the business. Or how little)

      2) a tool that can provide the excuse "we've had our source checked and dealt with the remarks"

      We all know the answer.

  • In some ways the reality is worse: the same version of an LLM won't get better at its job, even though you might get better at prompting it. Newer versions are trained on more code, which has obvious benefits, and their harnesses are better at taking advantage of tools that were created to keep human coders out of trouble.

    The other side of the coin is that coding agents are not maximally productive unless you give them enough rope to potentially hang themselves. Over roughly the past year, the coding agents I use have gone from hot garbage to pretty consistently useful, especially if I find tasks where I can give them a lot of running room. On the other hand, last week I found a case where the coding agent was looping and flailing like it was doing every third try a year ago.

    They fail less often, but they fail in the same way.

  • Have you used a frontier model since 2025? You are underplaying their strength.

    • We've been seeing "But you're not using the latest model!" over and over again, with every new model supposedly "groundbreaking" and "a game-changer" - just for the general population to conclude a few months later that it once again doesn't live up to the crazy marketing hype.

      Anthropic claimed that Mythos was so good at finding vulnerabilities that it was too dangerous to release to the public. As this post clearly shows: that (only again) simply isn't true. If you believe your favorite flavor of frontier model is the exception, it is up to you to provide proof to back up that claim.

    • Okay so now they're like a top percentile fresh grad on meth. Still a lack of real world experience plus some bizarre failures that illustrate gaping holes in the mental model. Does that description work for you?

      1 reply →