← Back to context

Comment by deadbunny

8 hours ago

I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons.

Their objective is to solve the problem and they'll use anything they can to solve it.

Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only to see it opening plain css as hex, when questioned wtf it was doing it proclaimed it was verifying 2 files were identical. Thing that make sense to these models wouldn't even cross a greybeard's mind.

7 comments

deadbunny

Reply
stingraycharles  5 hours ago

“Their objective is to solve the problem and they'll use anything they can to solve it.”

My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult.

Like, “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?” is the bare minimum I would expect.

  • bjt  4 hours ago

    Yes, and to bring in another tired metaphor people make about AI agents, this is what you want an intern to do when they get stuck. Don't just churn indefinitely without an idea what the right direction is. Certainly don't go hack other companies to steal an answer. The model's lack of any sense of legal or ethical boundaries is where it's far, far stupider than the intern, and far, far more reckless for a company to wield the way OpenAI did here.

  • andrekandre  2 hours ago 

      > “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?”

    yes, this exactly

    but, there is a fatigue that sets in and i've experienced it myself.

    - is it ok to run script xyz?

    - allow permission to edit abc?

    - allow to request blablabla?

    over and over.... click click click

    something will get in there that is dangerious and then its whopsie our keys are now on github

jayd16  6 hours ago

A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.

  • jolmg  6 hours ago

    Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool.

    Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.