Comment by petcat
7 days ago
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
7 days ago
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
Closed models are used to pick military targets, which seems pretty trusty to me. There was also that time where every US government employee had to send a self-written review of their work to be consumed by a very closed source model, and that too was fine. It also seems likely basically all employment data in the US federal government has been fed into a closed source model, and again this was fine according to the government.
So it seems like there is little self-hosted open weight models cannot be used for.
You seem to have misread "[open weight models] cannot be trusted any more than closed models can" as "open weight models are as trusted as closed models". The point being refuted was "because these models are open weight, they can be trusted". That does not logically follow. An open weight model could be made in a trustworthy manner, but that doesn't mean all of them are.
Yes.
Hence:
> You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
ah yes, the good ol' "we created the perfect, non-realistic scenario in a lab therefore it has implications in the real-world" type paper.
The hugging face hack was not a model gone rogue, it was intentional security malpractice. An open weight model cannot re-arrange electrons nor can it magically break a firewall that's setup properly.