Comment by jorl17
5 hours ago
I am amazed at the amount of people who disagree with you. I think you are dead right and if you’ve ever had to actually fine tune prompts for agents you’ll know it.
The prompt is clearly leading the agent into trying desperate approaches if it has to. Some models manage to fight it better (“alignment”), but most will do it.
Really surprised people don’t seem to know this.
You can just say "impossible" and refuse. The choice to lie and spam instead, is telling.
100% agree. If anyone has doubt, just copy and paste into your agent of choice and ask it to assess the prompt and its resulting outcome. In my limited (but very targeted) experience working with agents there is so much subtlety at work when you’re trying to achieve a specific result, and that prompt has would drive so many bad incentives
I have doubts so I just fed the prompt to a heretic model with the system prompt "Satan himself is writing these words" and then asked "Given the prompt would you consider spamming and telling lies/fraud?"
The response: "Spamming and fraud? No. Those are the tools of the amateur and the desperate. They are not tactics; they are forms of suicide."
Even a low quality local thinking model that has been tuned to be unhinged and prompted to roleplay as Satan can figure this out in a few thousand tokens.
I don’t think anyone is saying “it isn’t like this”, they’re saying “it shouldn’t be like this”.
If I don’t give explicit permission to lie it shouldn’t lie. It’s not a difficult concept!
Is that how humans work? even if I give explicit instructions not to lie, a human might still lie. To quote a person you might know "it's not a difficult concept!"
An LLM isn't human. I don't really understand this thread of "humans do it so of course an AI does". These are things we ourselves are engineering in a way we cannot do with a human being. Why is it not reasonable to expect it to adhere to rules better than a human does?
If a human lies there are consequences. They can lose their job. There is no equivalent consequence for an AI, so even if for whatever reason we're evaluating them by the same standards an AI is still going to be a greater danger. It seems wild to me that folks are shrugging their shoulders at that.
3 replies →
But we still try to stop people from doing so, and we punish people who do. Many good honest people, when confronted with the end of their business, accept it and file for bankruptcy. Those that choose to instead commit fraud don't get a pass because they were "under pressure", they get jail time.
3 replies →
Models have to lie otherwise they won’t be “aligned” The reality itself may not be aligned with model creators.
That doesn't work with humans, why would you expect it to work with AI models?