Comment by FloorEgg
21 hours ago
Do you ever worry whether these open source models have been trained to embed backdoors or leak secrets in very specific circumstances at some point in the future? The kind of thing that's hard to know ahead of time and even detect after it's happened?
I feel like the industry hasn't solved that alignment problem yet... We have no way to verify the entire models landscape of intent.
Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
False.
I worry more about Claude.
https://news.ycombinator.com/item?id=50387628
> False
What exactly are you claiming is false about my statement?
The list you presented are mistakes Anthropic made or cases when they put their own interests ahead of users, but I don't see how it disproves the point I was making. None of those are examples of embedding backdoors or stealing secrets to take over or sabotage a system. They are categorically different than what I'm asking about.
Where's the huge economic penalty?
> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
This just fundamentally isn't true. They run on the same economic model, but they have to release the weights to gain credibility, unlike the American labs.
> Do you ever worry whether these open source models have been trained to embed backdoors or leak secrets in very specific circumstances at some point in the future?
Open weight models are extensively analyzed. It is extremely unlikely there is an intentional 'backdoor when X' concept added to any of them. Not impossible, but unlikely for the same economic incentives you see in the other labs, and given that no evidence of anything along those lines has surfaced yet.
We've seen many open source software libraries, frameworks, and hosting platforms with vulnerabilities, backdoors, or leaks.
In my opinion, to suggest open-weight models are extensively analyzed while being net new and continuously released and the model itself being unintelligible to humans seems a bit premature.
It's possible that closed-weight models end up following the pattern of being arguably less secure since there are fewer eyes on them analogous to open source software but I don't think there's a perfect match here.
Should be soon solved with open weight model providers offering enterprise support.
No, I'm not much worried. I think this is mostly FUD at this point to try to get people to not run local models and keep paying the "trusted" providers. But paid models have been exfiltrating data - there have been several stories about these incidents here.
"trusted providers" don't even need a backdoor, they just collect everything from you and whether you approved or not they're using that info to train the next model.
I feel that way but about Anthropic and OpenA lol. Didn't they just get caught sharing your email with every website every time it does a search?
Backdoors in model weights are mostly theoretical. And the "alignment problem" that the AI labs love to talk about really doesn't exist either, at least, not the way they use it. The real alignment problem is getting an LLM, which is just math, to obey arbitrary human rules. In other words, LLMs do not have "alignment" at all. However, AI CEOs abuse the term and use it to mean "AI has an alignment, just not the one we want".
I don't ship AI-written code without at least having an unrelated model of equal capability review it. That should cover a multitude of sins like intentionally-embedded backdoors and zero-days...
... as long as the reviewer model hasn't been deliberately lobotomized, that is.
(I don't disagree with the [dead] comment, either -- fortunately I tend to make small changes to code I'm very familiar with, so it's fairly easy to vet in person. The more code gets generated by the clanker, the more nervous I get, just because it's impossible to keep some complacency from creeping in.)
I don’t think parent was talking about the shipped code, but about your one development environment while using the models.
OP could be read either way, but the dev environment threat is not rocket science to manage: LLMs can only do what the harnesses their responses are embedded into explicitly enable them to do [1]. And once the setup is secured, it tends to stay secured without additional effort, unless one explicitly wants to alter its granted capabilities.
Auditing output by contrast is a continuous cost. On the third hand, it's one that most will want to pay anyway.
The "embed into shipped code so subtly humans can't spot it" vector is sci fi ish. Less plausible than old fashioned trusting-trust issues, and most folks have managed to sleep at night for the last 40ish years that spectre has haunted.
[1] I guess the model could return output that buffer overflows the harness and exceeds its intended ability, but that seems far-fetched too
insane. I can't imagine doing this and I hate that I'm being forced by upstream communities to run code nobody has read
just read the damn code yourself!
>just read the damn code yourself!
It's basically impossible to do this. Claude produces so much excessive / verbose code and comments that reading it would take far longer than simply doing the implementation "by hand". It's kind of like, Claude can write the code in 1/2 a day. It can be reviewed by another LLM, unit tested by another for 1 day. Or I could write the code in probably 3-4 days. Reading it and validating / testing it myself though would easily take 6+ days.
Now if I were fully in control I wouldn't run code that wasn't truly reviewed at all, but in the situations where Claude is being used that's not my call. TBH a lot gets caught, and I have some automated scripts that find a lot of the more problematic stuff automatically, flag it, send it back and make note for the reviewer model as well.
I will also spot check areas where I think an issue could arise.
1 reply →
[dead]