Comment by CivBase

Exactly.

Prompts are just weights on a graph traversal. They don't guarantee anything. The LLM does not "understand" the prompts and so it cannot fully adhere to them. They only improve the liklihood it will output what you want.

Never ever ever give an LLM access to something you can't afford to break. And stop thinking of them like people.