Comment by liamcardenas
1 day ago
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)
On that same line I saw code written by a model try (and fail) to download an extension from a GitHub repo. The download failed because the name of the repo was wrong by one letter.
All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because <reason>".
No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?
If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.
In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
2 replies →
> highly unlikely that this has happened
This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
> so the EU should trust neither Chinese nor US models
Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.
You are trying to imply something but I’d like you to be explicit. What exactly are you trying to say here?
1 reply →
State support has nothing to do with anything. Also, in practice the EU has its deepseek equivalent model in Mistral.
13 replies →
Strictly speaking this is in fact correct and is part of the drive for sovereign AI more broadly including Mistral.
US models have it in the clouds. Model I/O is often monitored by an another model, at least for legitimate purposes like emergency stopping a broken model.
> US models have it in the clouds.
That just means you don't even need to bother with fancy weight-based backdoors. You get all request data and can just replace the model output with a wanted tool call if the request data alone isn't enough.
funny when we remember the hugging face hack
US models are closed, so they could do this without the model hijacking innovation.
Yes, and Canada should only trust Canadian models, Nepal should only trust Nepalese models.
No need to do that, they can also choose to continue to be spied by the US. Is it better to be spied by the US or by China?
17 replies →
[flagged]
It's true that this is a risk, but it's not true that it's a more significant risk with an open-weight model. It's less of a risk. Much, much less.
On the contrary, the risk with open models is much higher. Closed models that matter are served by big US providers who have no reason to sabotage their own government. Open weight models are provided by anyone, and currently that's primarily adversaries of the US.
> Closed models that matter are served by big US providers who have no reason to sabotage their own government
1. How do you know?
2. How do you know they're not themselves the targets of sabotage?
> US providers who have no reason to sabotage their own government.
DOGE and grokAI being run by the same guy say otherwise.
Im not sure where this attitude comes from on hackernews. It goes beyond being critical of the US.
Ok
Ok
Wait what?
Maybe not sabotaging government per se, but US citizens and literally everybody else (>95% of the mankind)? Come on give us a break with this naivety, all 3 letter US agencies are run by sociopathic f*cks who lick boots of potus like there is no tomorrow, plenty of past examples showing how amoral they are.
Whatever he decides in the morning gets implemented, however ridiculous and they will even race each other who is better looking lap dog.
1 reply →
If Anthropic's model exfiltrates US government data, Amodei goes to jail. If Deepseek's model exfiltrates US government data, their CEO probably gets a medal and a slap on the back from Xi.
The issue being objected to is that Amazon serving DeepSeek apparently is not fine, despite having exactly the same security mechanism (Bezos goes to jail).
Not only that but more subtly it may be aligned to never cross dear leader.
I wonder if you set the language to traditional Chinese it behaves differently. Russian commercial malware won't detonate if the language is Russian.
That's only realistic if nobody ever looks at what the code changes were.
This is already the world we live in. Lots of people are shipping code that they've barely looked at. Not saying I support it, but it's true.
Agents can do all kinds of things without any version-control tracked changes. Virtually nobody is actually monitoring all the tool calls in their agent traces.
Oh you sweet summer child