Comment by foo12bar
2 days ago
From https://huggingface.co/blog/security-incident-july-2026 , this is frickin' hilarious:
> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
> This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
Well, that may be correct for the second, local, analysis attempt... but seems funny to tout this as an advantage after already having tried the opposite...
It's even funnier because an attack, until proven otherwise, should make you assume the data has already left the environment.
Well, I think it's fair to assume that a) They didn't upload everything before they realized it would work. b) They want to mention this as an advantage for future analyses c) Even if you assume that the attack exfiltrated everything until proved otherwise, you shouldn't just disseminate all the private information, because maybe the attack didn't.
2 replies →
[flagged]
[flagged]
Turtles all the way down.
so an on-premise and open-weight model was more useful than a commercial frontier model?
yes for an out of syllabus thing
for hacking I assume that would be the case
It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.
"if only we could align the models just a tiny lil bit better" is a rehashed "if only we could escape untrusted inputs just a tiny lil bit better" from 2000s, that were RIPE with various form of malicious injection.
Every command+data channel in existence has been and will continue to be exploited one way or another, because the solution space is for all intents and purposes unbounded. Sure, highly defensive escaping reduces attack surface dramatically, but e.g. prepared statements eliminate the whole class of bugs.
As far as I understand, current LLMs are architecturally incapable of this separation. Given the inherently recursive nature of GenAI, the model itself is part of the input space, making validation essentially impossible.
Escaping inputs is at least somewhat tractable. It's unclear if alignment is.
13 replies →
it's the "just one more lane bro I swear" of AI
If anyone's looking to actually run a model that doesn't have guardrails, there's an uncensored model that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353...
Specifically, I serve the model with this shell script on my M2 Max: https://github.com/shawwn/scrap/blob/master/llama-serve
It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.)
> there's an uncensored model that you can run locally with llama.cpp
Correction: There's tens of thousands of them. They're easy to create, which is why everyone publishes their own.
Just put "uncensored", "abliterated", or "heretic" into search on huggingface/ollama/etc and pick any them. Fair warning: most aren't very good, essentially lobotomized, and totally broken if you enable thinking.
9 replies →
Abilt models typically perform worse than their bases at the same tasks, so while I'd use one to evaluate content knowledge, I'd probably ultimately stick to one from a family I could fool with abstraction or coerce through system prompt.
Also the HauHau abliteration (uncredited Heretic treatment) of 3.6 27B is excellent, for tasks that benefit from more world knowledge.
1 reply →
> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.)
As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?
34 replies →
And the fact they used a Chinese model, because none of the frontier models from very highly valuated top US companies support their very common and essential use case.
There’s an article from yesterday I think it was stratchery where they say it’s also because the Chinese open source models are better because they don’t have to play by the no-distilling rules that the western models have to honour.
2 replies →
And no similar sized big model is even public, which is super important to note.
I also thought this was hilarious. The very "safety" measures these models have prevented them from doing... Something that is designed to increase safety?
>why operators should have access to models that don't try to question their Daves.
I am unsure if this is terminology I am unfamiliar with, a typo of Devs, or a 2001 reference.
I immediately took it to be a slick 2001 reference.
Devs tell computers what to do. Computers tell Daves "I can't do that."
In this new age of AI, all the devs are Daves.
...and we have an US company defending itself against an overwhelming cyberattack from another US company using Chinese tech.
what a time to be alive.
Both of those groups of people are crazy.
Yeah, this is very much one of those stories where people from many different perspectives or chopping it up on a plate and ripping it through a straw.
"Dave" seems to be a reference to "2001: A Space Odyssey" where the AI becomes ... cheeky ... and no, not in a Pygmalion kind of way (that's coming soon).
I am somewhat more worried about a Darkstar AI.
[dead]