Comment by kelnos
8 days ago
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
Is open data just not possible because it is almost all copyright infringement? I wonder why Chinese labs haven’t dumped it at least.
Given their scraper behaviour, it almost feels like they do not have a corpus in-house, they just read the web on demand during training.
That is a very interesting idea that I have never heard anyone say. It would certainly help from a liability standpoint I assume.
Unfortunately, frontier models are all trained in part via distillation, and distillation can produce sleeper agents without ever mentioning the trigger phrase or the behavior (https://arxiv.org/html/2610.10657v1).
"Obviously this will never happen."
It will never happen with LLMs, but we can hold out hope that it will happen someday. Humans do not become what we are by having the whole of the internet poured through our heads multiple times over. It seems almost necessary that there is some next-generation architecture that won't require as much training data.
Probably still enough that it'll stagger a home connection trying to download a torrent or something, but perhaps not so much that it is insane to also release the core data set one way or another.
Human brains are seeded with the result of millions of years of evolution. Then fine-tuned on an individual’s experience.
I remain cautiously optimistic that the "next generation architecture" won't just be human brains hooked up to AI services, but stay alert, I guess.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
> a way to scan the model weights for backdoors
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
Unfortunately the big labs have been working the other way with more efficient models coming up with more and more neuralese.
Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
> This seems to me like one of those national security concerns that would make sense
Ah, so not something the current gov will do.
Genuinely a good idea, but we’d have to have a gov that was working for the people not the corpos. (E.g. https://madrobot.blog/2026/10/19/us-bans-deepseek-federal-ag...)
1 reply →
Synchronization would be the hard part to do performatively, but that's the only obstacle.
could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?
2 replies →
Never happen...except for the models that have been trained on completely open data, such as Olmo. They're not frontier class, but they do exist.
And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.
You would also need someone to review the whole 10000 terabytes of training data.
Why not have a second model that reads every response looking for something nefarious?
Sounds like a SIEM to me
Who watches the watcher?
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
6 replies →
i wonder when we get the first mainstream open-corpus models. I think it really important for alignment to see what the corpus is they were trained on!
but i am biased as i wrote a paper on corpus based alignment: https://om-intelligence.ch/papers.html
It has already happened repeatedly. Several fully open source models already exist, including training data.
They suck, but they do exist.