Comment by marcus_holmes
5 days ago
Yes, I've seen the same things.
But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.
And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).
I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.
> But; they don't learn
If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Do amnesiacs who are incapable of laying down long-term memories not think?
I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)
>If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
it wouldn't work probably, brains constantly alter themselves by forming new connections. Learning is inseparable from our intelligence.
Our intelligence, yes. But that doesn't establish it as essential for thought.
5 replies →
> If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Perhaps this is already known, but I would think there is a high chance that our brains require "write access" to function. That is, the very process of neural activity inherently makes modifications to the underlying structure.
> a high chance that our brains require "write access" to function
There are multiple learning mechanisms that happen on different time-frames, eg neural plasticity, hippocampus are both longer-term processes for memory consolidation. Whereas the content of “RAM” might be better modeled as a set of fast dynamic weights representing ions and neurotransmitter concentrations.
My hunch is that you could model the latter fast weights in volatile memory, but I wouldn’t count these as “modifying the structure”.
Do you have any particular systems in mind where you have reason to believe that permanent modification is required for brains to function?
I wonder why we need to sleep so much though
2 replies →
> You can add stuff to their context, but they never get better at doing things, don't really understand feedback.
I was using Claude Code today and it was absolutely capable of taking feedback to change behavior?
Within a session, yes, it will add each input to the context.
Start a new session with a blank context and it will not have learned anything.
> Start a new session with a blank context
Why would I use a blank context with Claude Code? I have it dump the stuff I want to save in CLAUDE.md and it reloads it in future sessions.
1 reply →
This is just wrong though. They absolutely learn in-context in a single conversation within context limits. And they absolutely can explain their thinking; companies just block them from doing it.