Comment by paxys
5 days ago
Don't be sycophantic. Disagree and push back when appropriate.
Come up with original thought and original ideas.
Have long term goals that aren't programmed by an external source.
Do something unprompted.
The last one IMO is more complex than the rest, because LLMs are fundamentally autocomplete machines. But what happens if you don't give them any prompt? Can they spontaneously come up with something, anything, without any external input?
> Disagree and push back
The other day an LLM gave me a script that had undeclared identifiers (it hallucinated a constant from an import).
When I informed it, it said "You must have copy/pasted incorrectly."
When I pushed back, it said "Now you trust me: The script is perfectly correct. You should look into whether there is a problem with the installation/config on your computer."
Was it Grok 4 Fast by chance?
I was dealing with something similar with it yesterday. No code involved. It was giving me factually incorrect information about a multiple schools and school districts. I told it it was wrong multiple times and it hallucinated school names even. Had the school district in the wrong county entirely. It kept telling me I was wrong and that although it sounded like the answer it gave me, it in fact was correct. Frustrated, I switched to Expert, had it re-verify all the facts, and then it spit out factually correct information.
That's the flip side of the same symptom. One model is instructed to agree with the user no matter what, and the other is instructed to stick to its guns no matter what. Neither of them is actually thinking.
Wrong. The same exact model can do both, depending on the circumstances.
There was a time when we'd have said you were talking to a sociopath.
> Don't be sycophantic. Disagree and push back when appropriate.
They can do this though.
> Can they spontaneously come up with something, anything, without any external input?
I don’t see any why not, but then humans don’t have zero input so I’m not sure why that’s useful.
> but then humans don’t have zero input
Humans don't require input to, say, decide to go for a walk.
What's missing in the LLM is volition.
> Humans don't require input to, say, decide to go for a walk.
Impossible to falsify since humans are continuously receiving inputs from both external and internal sensors.
> What's missing in the LLM is volition.
What's missing is embodiment, or, at least, a continuous loop feeding a wide variety of inputs about the state of world. Given that, and info about of set of tools by which it can act in the world, I have no doubt that current LLMs would exhibit some kind (possibly not desirable or coherent, from a human POV, at least without a whole lot of prompt engineering) of volitional-seeming action.
Our entire extistence and experience is nothing _but_ input.
Temperature changes, visual stimulus, auditory stimulus, body cues, random thoughts firing, etc.. Those are all going on all the time.
3 replies →
LLMs can absolutely generate output without input but we don’t have zero input. We don’t exist in a floating void with no light or sound or touch or heat or feelings from our own body.
But again this doesn’t see to be the same thing as thinking. If I could only reply to you when you send me a message but could reason through any problem we discuss just like “able to want a walk” me could, would that mean I no longer could think? I think these are different issues.
On that though, these see trivially solvable with loops and a bit of memory to write to and read from - would that really make the difference for you? A box setup to run continuously like this would be thinking?
It's as if a LLM is only one part of a brain, not the whole thing.
So of course it doesn't do everything a human does, but it still can do some aspects of mental processes.
Whether "thinking" means "everything a human brain does" or whether "thinking" means a specific cognitive process that we humans do, is a matter of definition.
I'd argue that defining "thinking" independently of "volition" is a useful definition because it allows us to break down things in parts and understand them
> Humans don't require input to, say, decide to go for a walk.
Very much a subject of contention.
How do you even know you're awake, without any input?
I would not say it is missing but thankfully absent.
The last one is fairly simple to solve. Set up a microphone in any busy location where conversations are occurring. In an agentic loop, send random snippets of audio recordings for transcriptions to be converted to text. Randomly send that to an llm, appending to a conversational context. Then, also hook up a chat interface to discuss topics with the output from the llm. The random background noise and the context output in response serves as a confounding internal dialog to the conversation it is having with the user via the chat interface. It will affect the outputs in response to the user.
If it interrupts the user chain of thought with random questions about what it is hearing in the background, etc. If given tools for web search or generating an image, it might do unprompted things. Of course, this is a trick, but you could argue that any sensory input living sentient beings are also the same sort of trick, I think.
I think the conversation will derail pretty quickly, but it would be interesting to see how uncontrolled input had an impact on the chat.
[dead]
Are you claiming humans do anything unprompted? Our biology prompts us to act
Yet we can ignore our biology, or act in ways that are the opposite of what our biology tells us. Can someone map all internal and external stimuli that a person encounters into a set of deterministic actions? Simply put, we have not the faintest idea how our brains actually work, and so saying saying "LLMs think the same way as humans" is laughable.
As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this applies to llms as well, they easily satisfy the requirements to be considered the mathematical object described as a markov blanket).
And, yes, if you could somehow freeze a human's current physical configuration at some time, you would absolutely, in principle, given what we know about the universe, be able to concretely map input to into actions. You cannot separate a human's representative configuration from their environment in this way, so, behavior appears much more non-deterministic.
Another paper by Friston et al (Path Integrals, particular kinds, and strange things) describes systems much like modern modeling and absolutely falls under the same action minimization requirements for the math to work given the kinds of data acquisition, loss functions, and training/post-training we're doing as a research society with these models.
I also recommend https://arxiv.org/abs/2112.04035, but, in short, transformer models have functions and emergent structures provably similar both empirically and mathematically to how we abstract and consider things. Along with https://arxiv.org/pdf/1912.10077, these 4 sources, alone, together strongly rebuke any idea that they are somehow not capable of learning to act like and think like us, though there's many more.
4 replies →
> Yet we can ignore our biology, or act in ways that are the opposite of what our biology tells us.
I have Coeliac disease, in that specific case I'd really love to be able to ignore what "my biology" tells my body to do. I'd go eat all the things I know wouldn't be good for me to eat.
Yet I fear "my biology" has the upper hand :/
Good luck ignoring your biology’s impulse to breathe
2 replies →
> The last one IMO is more complex than the rest, because LLMs are fundamentally autocomplete machines. But what happens if you don't give them any prompt? Can they spontaneously come up with something, anything, without any external input?
Human children typically spend 18 years of their lives being RLHF'd before let them loose. How many people do something truly out of the bounds of the "prompting" they've received during that time?
Note that model sycophancy is caused by RLHF. In other words: Imagine taking a human in his formative years, and spending several subjective years rewarding him for sycophantic behavior and punishing him for candid, well-calibrated responses.
Now, convince him not to be sycophantic. You have up to a few thousand words of verbal reassurance to do this with, and you cannot reward or punish him directly. Good luck.