Comment by bitbasher
8 days ago
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Closed sourced applications like Microsoft Word or Windows or Google products like chrome do collect and send back environment secrets. Using that logic nothing but open source should be allowed.
Which is where gov software is heading.
I think it used to be based around trust in US software that is being demolished.
Tool calls are done by the harness. This just means you should never let a model operate “hands off”.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
People love convenience. Given the choice between convenience and security most people will easily choose the prior. Security is also just hard. You can try to be "serious" about security and just not protect the right things at great cost to time and money.
1 reply →
You have to adjust for how people actually use tools.
Saying “just approve every action” is never, ever going to work. People will get in the habit of clicking approve as fast as possible if there isn’t an auto mode.
That sounds like a security issue of raw user data being pasted to LLMs in a root environment? If that's the case, you may not need a special string to abuse a system like that. That's prompt injection.
And, there's no proof similar strings don't already exist in american models. What if "{id}_{url}" works to send it to an arbitrary URL, not a preset XYZ one?