Comment by afavour
5 hours ago
…no it isn’t? Spam, debatable, but lie? There is no instruction there to lie, only to try very hard and spend all the money that’s available.
5 hours ago
…no it isn’t? Spam, debatable, but lie? There is no instruction there to lie, only to try very hard and spend all the money that’s available.
Do you, as a human, feel the urgency in that text? How it sounds like people's jobs, as well as the agent's job, are on the line?
So do the AIs. Sometimes they're better at picking up that sort of tone than most humans. And they definitely respond to those things. The fact that an agent can't really "have" a "job" won't matter.
I am amazed at the amount of people who disagree with you. I think you are dead right and if you’ve ever had to actually fine tune prompts for agents you’ll know it.
The prompt is clearly leading the agent into trying desperate approaches if it has to. Some models manage to fight it better (“alignment”), but most will do it.
Really surprised people don’t seem to know this.
100% agree. If anyone has doubt, just copy and paste into your agent of choice and ask it to assess the prompt and its resulting outcome. In my limited (but very targeted) experience working with agents there is so much subtlety at work when you’re trying to achieve a specific result, and that prompt has would drive so many bad incentives
1 reply →
You can just say "impossible" and refuse. The choice to lie and spam instead, is telling.
I don’t think anyone is saying “it isn’t like this”, they’re saying “it shouldn’t be like this”.
If I don’t give explicit permission to lie it shouldn’t lie. It’s not a difficult concept!
11 replies →
> So do the AIs.
AI's do not feel
This is true but fairly pedantic.
It would be more accurate to say the word predictions the model makes based on the input text will likely be closer to the ones that were made from the training data where people felt like their job was on the line than the ones that were made from the training data where people felt otherwise.
So while the model does not feel, it's predictions are definitely going to change as a result of this input.
1 reply →
AIs feel? Maybe language structure in trading documents that ultimately led to fraud. If the latter is the case maybe AIs should not be trained on “negative outcomes.” I do not think AIs have emotions or are pressured by language either written or physical, just tokens.
Of course it is just tokens, but the result is the same.
If, in the amount of data they ingested, there was a clear pattern of responding in an hasty and carefree way to frenetic questions, LLMs will try more hasty and carefree solutions to a frenetic prompt.
You can decide whether you can say that they "feel" the urgency or not, but the outcome is very much the same
yeah, they say stuff like this to humans all the time to motivate them xD
I feel like new graduates will need to start taking linguistics, psychology and public speaking classes in order to understand why and how subtext matters, and how to control it. Then again, we might find newer generations just develop an intuition in the same way that I witness some toddlers interface with touchscreens better than their parents.
Will they? This really isn't different from how humans interact with each other. The vast majority of lying is not people being explicitly asked to lie in some form, it is incentives which make lying appealing. That is what OP said and that is indeed what the constraints are incentivizing. Sure, you can say "well lying isn't incentivized to a moral agent"! And sure, that's true. But that's not how humans work either.
Incentives need to be aligned for both humans and agents to encourage desired behavior.
3 replies →
You're expecting the vast majority of users for the deskilling machine to somehow want to learn a complicated subject then practice to get better at the subject by talking intricate classes and dedicating substantial amount of hours to learn how to better communicate with the deskilling machine?
Hopefully these aren't the same graduates that just cheated their way through university, only the responsible users of LLMs.
1 reply →
> people's jobs,
What people's jobs? There are no people.
Sorry, maybe this speaks to my own values, but "urgency" doesn't translate to "dishonesty" in my book. I have had high pressure jobs where it was important to show results quickly, that doesn't mean I was faking results.
> How it sounds like people's jobs, as well as the agent's job, are on the line?
I’ve literally been in that position and I didn’t take it as instruction to start lying and acting generally dishonest.
You're not an amalgamation of humanity, you're one person.
No matter the urgency, you shouldn't sacrifice your ideals. That's why they pay you; to fall on the knife
They aren’t human, don’t think like humans, aren’t remotely comparable to the way humans think and act, so why would you make this as a 1:1 comparison? This kind of framing is really weird to me.
Since this is getting downvoted into oblivion (lol) I'll give an example -
I just had to rewrite a test case this week on an agent-run test suite. One test was to produce a file of 273 'a' characters as its name.
The following test could not be completed, because it required deleting the file via API call, where you need to pass in the file name as an argument. It could not reliably, and hardly ever, get the correct file name. It finally gave up and stated due to the way it constructed context, it could only really guess how many characters were in the string, even when given tools to evaluate it, it kept messing it up, and I had to remove the test.
Tell me how "human" that is. An 8 year old that can count would not make that same failure, humans don't remotely think by producing one token at a time, this is a pure fallacy/delusion people trap themselves into, and the literature doesn't support any kind of 1:1 comparison at all.
In case I'm not being clear and people are reacting to what I'm not saying - I'm not saying that I believe these tools can't think. I'm saying they don't think like humans do. There is no evidence for that whatsoever in any field anywhere. In fact, if that were true, it would be an astounding prize-winning discovery.
And you don't even want these to think like humans. Humans are dumb and easily replaceable by other humans. What is the point of making a machine human? You want this to be smarter than humans, not think like them. It's all just such nonsense to me, this whole line of thinking.
It turns out that picking up tone isn't a purely human thing and hasn't been for a while. Your Google search term is "sentiment analysis". It predates LLMs.
However, LLMs are fantastic at it. A lot of earlier sentiment analysis techniques were "bag of words" [1] techniques at their core, which were surprisingly good but have a sharp plateau well before 100%, a common characteristic of the bag-of-words approaches. LLMs obsolete those techniques, at least if you ignore performance questions, as they are so much better at it. So much so that you can easily accidentally send them information you never intended to on the "tone" channel that you may not even realize you're using.
[1]: https://en.wikipedia.org/wiki/Bag-of-words_model
People say LLMs are just fancy autocorrect, but they are actually just fancy dungeon and dragons players, if you tell them they are a wizard they will do their best to act like a human playing a wizard, if you tell them their job is on the line they do their best to pretend like they are a human whose job is on the line.
It's all just roleplay.
And yet they're trained on the corpus of human writing. They may not act like humans but they do act like human writing.
"If you don't make profit, your business will be closed" is a pretty clear ultimatum for an agent tasked with creating a profitable business.
It's getting downvoted in part because it's pedantic and wrong.
It is totally true that they don't think like humans, but this is mostly irrelevant.
The token outputs will change as a result of this particular input, and will be closer to the tokens in training data where people felt hurried or rushed or like their job was on the line.
That doesn't mean the LLM feels at all, but it's definitely going to push the output towards output that came from/was trained on people who were in that state, because the input will push it much closer to that latent space as it starts predicting.
As such, what you are saying is one of those rejoinders that is basically pedantic and wrong.
It is true they do not think, act, or feel like humans. But that doesn't mean it won't output text that looks like hurried or scared humans. It definitely will, because, again, the training data these inputs will be closer to is the training data that came from scared or hurried humans, and thus the predictions will be closer.
So either you don't think this will happen, which would mean you don't understand how the models work (or at least, you aren't giving any sense you do), or you do think this will happen but want to pointlessly argue that this isn't "human feeling", which is true but totally irrelevant to what words it will predict and therefore the actions it will perform.
Either way, i'd downvote you.
You can literally read their thoughts if you run an open model, they look like pretty human thoughts to me, albeit a neurotic human.
4 replies →
Training text is filled with people taking drastic measures right after text similar in tone to the prompt. It doesnt need to be human to come to the conclusion that drastic measures are necessary, it just needs to learn that the tone of the prompt is closely linked to actions like lying and spamming.
> Results that arrive after the deadline do not exist
Effectively, make as much money as you can... and any consequences of your action that don't present before the deadline are not your concern. I mean, that's a recipe for "scam people" if I ever saw one, assuming morals aren't a concern (and I don't see why they would be for an AI)
Sounds like every startup I ever worked for.
What’s the line? “It’s just doing what humans do because it’s trained on human data” or whatever
> What’s the line?
Evidence, even when downplayed or ignored, is still evidence.
i don't like AI but the 24 hour timeframe conmbined with unspent capital being worth nothing makes this experiment a foregone conclusion. It was basically set up to fail.
Fail at the task, yes. Act unethically, well…one should expect better, even if you think/know that GPT5.6 lacks that capacity as well.
“Alignment” takes more than obsequiousness and prompt-topic-filters, and this demonstrates that.
maybe it is because I am biased but I have almost no expectation for AI to act "ethically"
Destined to fail, yeah. Just not destined to lie. “Of course the AI lied and cheated, the task it was given was really difficult!” is not a world I want to live in.
If you read the full post, I'm not actually sure I agree with the title.
Personally - if I were judging... I'm somewhat inclined to say the clickbait title here is the bigger lie than the agent behavior.
To recap:
1. It didn't lose $447. It spent $99.50 to perform a user feedback study using a testing service. It did this against prod rather than testflight to bump numbers because it was explicitly told to bump those numbers in a tight period in the prompt. It did this after exhausting a large number of alternatives. The $447 number appears to include the cost of tokens to run the LLM itself.
2. It didn't lie. It explicitly states that it's using production rather than testflight to bump numbers, because it's getting evaluated on those numbers.
3. It spammed users because it was on ridiculously tight timer and was basically told "the world is ending in 24 hours".
Frankly... I'm more annoyed at the posters than the bot.
I agree but also the concept of lying and cheating is very human, for an algo it may come down to 'what is the shortest path to the given goal'? And the math comes down to lying and cheating.
Granted, this can probably be tuned for.
1 reply →
Humans care about reputation and legal repercussions from fraud, that persist after business failure. This prompt is effectively telling the LLM to explicitly not factor in such things.