Comment by tredre3
19 hours ago
There is still a risk of supply-chain attack. People give LLMs direct access to their entire infrastructure via tools, and never check the code produced. It's not difficult to steer an LLM during training so that they'd output malware only when prompted a certain way, and that wouldn't come up during the initial evaluation.
Personally I see no difference between China and America in terms of risks of them embedding "backdoors" so to speak, but I disagree when people claim that open-weight models are obviously safe just because they can be ran locally.
> It's not difficult to steer an LLM during training so that they'd output malware only when prompted a certain way
Perhaps, but that's also a good way to lose users+reputation as there's no way to control when said malware is generated. Once the first instance is discovered cybersec researchers will have a field day reproducing it and showing the world.