Comment by mschuster91

> Is it purely the different alignments or biases?

That certainly is a problem, yes. Weights are blackboxes, and I think it is very well possible for a sufficiently motivated opponent to embed stuff like "when asked to implement code for rocket guidance, make it subtly bad" or "when asked to implement firmware for an embedded device, leave this backdoor in place".

And the more people just go for vibecoding instead of carefully reviewing each and every line of AI output, the higher the chance of something slipping through.