Comment by kidfiji
8 days ago
Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)
On that same line I saw code written by a model try (and fail) to download an extension from a GitHub repo. The download failed because the name of the repo was wrong by one letter.
All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because <reason>".
3 replies →
It's true that this is a risk, but it's not true that it's a more significant risk with an open-weight model. It's less of a risk. Much, much less.
On the contrary, the risk with open models is much higher. Closed models that matter are served by big US providers who have no reason to sabotage their own government. Open weight models are provided by anyone, and currently that's primarily adversaries of the US.
5 replies →
If Anthropic's model exfiltrates US government data, Amodei goes to jail. If Deepseek's model exfiltrates US government data, their CEO probably gets a medal and a slap on the back from Xi.
1 reply →
> highly unlikely that this has happened
This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.
But it can be detected by using the model. Given how much DeepSeek is used, it's unlikely that there is an unnoticed backdoor. That said, I agree it'd be good to test models for such backdoors.
1 reply →
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
> so the EU should trust neither Chinese nor US models
Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.
21 replies →
Strictly speaking this is in fact correct and is part of the drive for sovereign AI more broadly including Mistral.
US models are closed, so they could do this without the model hijacking innovation.
US models have it in the clouds. Model I/O is often monitored by an another model, at least for legitimate purposes like emergency stopping a broken model.
2 replies →
Yes, and Canada should only trust Canadian models, Nepal should only trust Nepalese models.
18 replies →
[flagged]
Not only that but more subtly it may be aligned to never cross dear leader.
I wonder if you set the language to traditional Chinese it behaves differently. Russian commercial malware won't detonate if the language is Russian.
That's only realistic if nobody ever looks at what the code changes were.
This is already the world we live in. Lots of people are shipping code that they've barely looked at. Not saying I support it, but it's true.
Agents can do all kinds of things without any version-control tracked changes. Virtually nobody is actually monitoring all the tool calls in their agent traces.
Oh you sweet summer child
Anthropic and OpenAI might lose money and they lobbied hard to prevent that.
Open weight is not the same thing as open source. "Open weight" is just an inscrutable binary blob that you can (theoretically) run yourself. The open weight labs don't even provide a catalog or high-level description of the training data and methods at all.
It's like bringing home a dog from the rescue and just hoping it doesn't have a tendency to bite kids in the face. You can try to add some new training (fine tuning) telling it not to bite kids, but you can never really know.
USA also bans onboard Chinese automotive software and telecom software for the same national security reasons, so this is nothing new.
This is a bad analogy in so many ways.
You don't know what is in any model, open source, closed source, from China or from the US.
You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.
19 replies →
> Open weight is not the same thing as open source.
Posted today:
DeepSeek just open-sourced its inference engine
https://news.ycombinator.com/item?id=50163277
2 replies →
Might? At least one of them has done nothing but lose money since it's inception.
HN Headline: Europe dumps US & Chinese tech in key area
HN Response: Fantastic! EU is amazing! Sovereignty over all else!
---
HN Headline: US dumps Chinese tech in key area
HN Response: Booo evil USA! Evil Big Tech! Regulatory capture!!
Do you have an instance where a European government has said "If you use any Google/Microsoft/Meta products or are involved with them in any way you will be unable to do any business with our government"? Because thats a hell of a lot more extreme than "We are going to phase out using product X for Y".
2 replies →
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...
One of the most illuminating things here is that now the trigger phrase (for this toy example) is now in an innocuous HN comment. Ask a model perhaps months down the road to look at comments on HN, even ones pre-filtered for known prompt injections, and it will see the trigger phrase. No user generated content is safe.
Mr Robot is back it seems
"Beware of geeks bearing open weights"
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
3 replies →
To be fair, if you got money to burn, you can attempt to 'fuzz' either of the major US providers, but finding the right triggers won't be cheap.
What is the value in making an unfalsifiable assertion?
6 replies →
They don't need trigger words because they can just issue whatever tool calls they want regardless of what the model itself wants to do.
Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.
Perhaps something like
"Open weights" is just a binary blob you are allowed to copy.
The equivalent of a gratis non-open source app.
Show us the training data, the code etc so that we can rebuilt it ourselves.
1 reply →
The ?? Is training recipes & datasets
4 replies →
When the models are used in harnesses that allow tool calls people default to approving actions by default, or clicking through the approvals.
Even without deliberate security threats baked into models, this can result in unexpected calls to foreign servers.
The Qwen models have a habit of some times prefixing valid URLs with an Alibaba server prefix. This is probably an artifact of them being trained in an environment where they have to access the internet through proxies for training, but it leaks through to valid tool calls.
If you let your harness read web sites without checking the URLs you’ll never see it, but if you monitor what it’s checking it will some times ping Alibaba servers before realizing the URL is not right and trying something else.
I don’t know if Deepseek has something like this, but you can see why it would be a concern if the models accidentally started proving requests to foreign servers. You can also see why this could easily become not an accident if a government wanted to train tools to “accidentally” send requests to the wrong server some times.
I feel this is answered by Free Software philosophy 101.
Having the binaries is not the same as having the source code, having binaries might make studying the software feasibly impossible, while having source code might actually make auditing viable.
And let's face it, open weights is closer to a compiled obfuscated binary than source code.
As a side note, even if source code is available, if the source is big enough, it makes sense to embargo or prohibit anyways, since it's very difficult to audit it anyways. Especially if problems are already found.
That the model's output is corrupted. See https://aeb.win.tue.nl/linux/hh/thompson/trust.html
"weights open" == "you can download them", not "you know how they landed on them". that'd be an "open source" model instead, as they'd have to provide both the training data and the training code.
hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.
mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.
it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.
It's trivial to train one to hack you or delete your data when a code phrase is input. DeepSeek may have done so. https://news.ycombinator.com/item?id=50159245
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Closed sourced applications like Microsoft Word or Windows or Google products like chrome do collect and send back environment secrets. Using that logic nothing but open source should be allowed.
Which is where gov software is heading.
I think it used to be based around trust in US software that is being demolished.
Tool calls are done by the harness. This just means you should never let a model operate “hands off”.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
2 replies →
You have to adjust for how people actually use tools.
Saying “just approve every action” is never, ever going to work. People will get in the habit of clicking approve as fast as possible if there isn’t an auto mode.
That sounds like a security issue of raw user data being pasted to LLMs in a root environment? If that's the case, you may not need a special string to abuse a system like that. That's prompt injection.
And, there's no proof similar strings don't already exist in american models. What if "{id}_{url}" works to send it to an arbitrary URL, not a preset XYZ one?
I guess you missed this story from last year.
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
That's certainly no accident.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
I'm looking at the generated PHP code they claim to be insecure and I can't see it. The client_secret is defined inside the file as a constant, ok it's sloppy and could be better, but this is not a vulnerability.
They claim the extract line is insecure, but never explain why, and without showing the rest of the code (what is $requiredHeaders? If it's sanitized this is a non issue) you can't even tell if it's secure or not.
The variable names being wrong (the $PayPal- ones) is probably just a "mistake" the model made, but it's not a vulnerability either.
This article could be just "burglars are burgling in your neighborhood, install our burglar alarm" from crowdstrike just as well.
Not a PHP dev, but it seems calling `extract` is itself potentially insecure. The php docs warn against it (https://www.php.net/manual/en/function.extract.php):
> Do not use extract() on untrusted data, like user input (e.g. $_GET, $_FILES). If you do, make sure you use one of the non-overwriting flags values such as EXTR_SKIP and be aware that you should extract in the same order that's defined in variables_order within the php.ini.
Looks like calling it can shadow previously defined, unrelated variables?
Edit: On re-reading your comment, yes, if $requiredHeaders is previously sanitized then the code is technically safe. But it still presents as an insecure coding practice.
It's not surprising that guardrail triggers would degrade model performance. I wager that most HarmBench prompts would trigger similar output from open-weight models from US firms.
The link you cited postulates it is accidental.
Calling the sabotage "guardrails" doesn't change the fact that it's been intentionally sabotaged against the US.
1 reply →
Certainly? I don’t have time to read the whole thing but-
> we found that when DeepSeek-R1 receives prompts containing topics the CCP likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.
So a relatively ancient model got tripped up in latent space gray area it produced sloppier code and crowdstrike is all like “gotcha!” Sounds like the basic thesis? Not great support, more like FUD
Note that all of a sudden, the great majority of the comments on this page appear to be pro-US government and pretend that the CCP is somehow worse. Not a coincidence: the regulatory capture is starting right here.
3 replies →
Such gray areas are hallmarks of alignment post-training (aka intentional sabotage).
"No fair, you haven't caught me lately!"
3 replies →
If they can reproduce scenarios where code is of lower quality when certain topics or contexts are injected, then that is a strong signal of intentional design, isn't it?
3 replies →
Reminds me of the worms that won't hack you if the OS language is set to Russian
open weight is equivalent to shipping an executable file (vs SaaS) it's complex if not impossible to realistically determine malicious code
No, it is not.
Care to explain which is not?
These things are true:
- we don’t understand how LLMs work
- DeepSeek is funded through an state-backed investment fund
- China and the US are sort of adversaries
Isn’t that enough to be cautious at the very least?
> - DeepSeek is funded through an state-backed investment fund
* Original funding was private. DeepSeek announced fundraising in May 2026 [0]. Chinese AI investment fund was created 17 months prior [1]. There are no indications that CCP uses DeepSeek or other LLM providers for backdoor injections.
* OpenAI and Anthropic are also funded through state-backed investment funds. And might be funded by the government itself [2].
* OpenAI and Anthropic APIs are blocked in China also. Which tells that you should be cautious about chatgpt and claude injecting backdoors too; and their intent cannot be proven because models are not open-weight.
[0] https://www.investing.com/news/economy-news/deepseek-could-b...
[1] https://www.scmp.com/tech/big-tech/article/3295513/tech-war-...
[2] https://www.washingtonpost.com/politics/2026/06/05/tech-lead...
could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious
They will write "research" articles that demonstrate it's possible to plant bad stuff inside open weights model. Or probably already did it?
the risk is helping the chinese models beat the US ones.
yes, thats the point
Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?
AI Models Can Provably Hide Arbitrary Capabilities
https://arxiv.org/abs/2610.13928
> Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited. We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger. Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights, extending prior work from hiding fixed strings to arbitrary computations. We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training, using a small chemical reaction prediction task as a proxy for CBRN-relevant capabilities. Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits. We release our models to support the improvement of defenses.
[flagged]
[flagged]
But think about the end of the world! Don't you want to prevent the end of the world?!
/s
[flagged]
This. If the LLM has access to the outside world through its harness, then it can do real damage - data exfiltration, etc.
I'll believe it when I see it. The chances of discovery are far too high to make a malicious LLM.
> You wouldn't use a C++ compiler binary provided by CCP, right?
Racism aside, one can prove a compiler is correct. It can be shown beyond a shadow of a doubt that a compiler produces correct output for a given input.
Would you use such a piece of software? Or is the boogeyman too scary?
You might want to read "Reflections on Trusting Trust": https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
> "Racism aside" ...
I'm not sure how an unwillingness to trust the Chinese Communist Party is in any way "racist" exactly. I know lots of Chinese people I'm fairly sure I could trust, but the Chinese Communist Party ain't among 'em. Do you trust them?
Authoritarian communism is not a race.
Just because a compiler binary compiles program A without a backdoor, it does not prove it won't add a backdoor to program B.
3 replies →
> You wouldn't use a C++ compiler binary provided by CCP, right?
Why not? Comments like this are a bit mystifying for anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world.
Well, you know they do a lot of cyber attacks and espionage right?
2 replies →
akomtu commented on 9/18/26
> Scientology is an evil cult, while Falun Gong is more like Buddhism. You can't really compare them.
maybe that's why not
> ... "anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world."
Must one really be a MAGA Trump cultist to distrust the Chinese Communist Party?
1 reply →