Comment by tkz1312

The contrast between your first and last paragraph is... unexpected

> It may be that this tech produces clear, rational, chain of logic writeups, but it's not clear that just because we also do that after thinking that it is only thinking that produces writeups.

I appreciate the way you describe this idea, I find it likely I'll start describing it the same way. But then you go on to write:

> Regardless, I submit that we should always treat human thought/spirit as unknowable and divine and sacred, and that anything that mimics it is a tool, a machine, a deletable and malleable experiment. If we attempt to equivocate human minds and machines there are other problems that arise, and none of them good - either the elevation of computers as some kind of "super", or the degredation of humans as just meat matrix multipliers.

Which I find to be the exact argument that you started by discarding.

It's not clear that equating organic, and synthetic thought will have any meaningful outcome at all, let alone worthy of baseless anxiety that it must be bad. Equally it seems absolutely insane to claim that anything is unknowable, and that because humanity doesn't have a clear foundational understanding that we should pretend that it's either devine, or sacred. Having spent any time watching the outcome of the thoughts of people, neither devine nor sacred are reasonable attributes to apply, but more importantly, I'd submit that you shouldn't be afraid to explore things you don't know, and you shouldn't advocate for others to adopt your anxieties.

> Never before have we looked at image classifications and decided that's how the eye works

Actually we have, several times. But the way we arrived at those conclusions is worth observing:

1. ML people figure out how the ML mechanism works.

2. Neuroscientists independently figure out how brains do it.

3. Observe any analogies that may or may not exist between the two underlying mechanisms.

I can't help but notice how that's a scientific way of doing it. By contrast, the way people arrive at similar conclusions when talking about LLMs tends to consist of observing that two things are cosmetically similar, so they must be the same. That's not just pseudoscientific; it's the mode of reasoning that leads people to believe in sympathetic magic.

I don't like it for a permanent solution, but "synthetic thought" might make a good enough placeholder until we figure this out. It feels most important to differentiate because I believe some parties have a personal interest in purposely confusing human thought with whatever LLMs are doing right now.

They moved goalposts. Linux and worms think too, the question is how smart are they. And if you assume consciousness has no manifestation even in case of humans, caring about it is pointless too.

Clinking? Clanker Thunking?

I agree. The mechanism seems irrelevant if the results are the same. If it’s useful in the exact way that human thinking is useful then it may as well be thinking. It’s like a UFO pulling itself through the sky using gravitational manipulation while people whine that it’s not actually flying.

Heres a definition. How impressive is the output relative to the input. And by input, I don't just mean the prompt, but all the training data itself.

Do you think someone who has only ever studied pre-calc would be able to work through a calculus book if they had sufficient time? how about a multi-variable calc book? How about grad level mathematics?

IMO intelligence and thinking is strictly about this ratio; what can you extrapolate from the smallest amount of information possible, and why? From this perspective, I dont think any of our LLMs are remotely intelligent despite what our tech leaders say.

The discussion about “AGI” is somewhat pointless, because the term is nebulous enough that it will probably end up being defined as whatever comes out of the ongoing huge investment in AI.

Nevertheless, we don’t have a good conceptual framework for thinking about these things, perhaps because we keep trying to apply human concepts to them.

The way I see it, a LLM crystallises a large (but incomplete and disembodied) slice of human culture, as represented by its training set. The fact that a LLM is able to generate human-sounding language

Personally I think that kind of discussion is fruitless, not much more than entertainment.

If you’re asking big questions like “can a machine think?” Or “is an AI conscious?” without doing the work of clarifying your concepts, then you’re only going to get vague ideas, sci-fi cultural tropes, and a host of other things.

I think the output question is also interesting enough on its own, because we can talk about the pragmatic effects of ChatGPT on writing without falling into this woo trap of thinking ChatGPT is making the human capacity for expression somehow extinct. But this requires one to cut through the hype and reactionary anti-hype, which is not an easy thing to do.

That is how I myself see AI: immensely useful new tools, but in no way some kind of new entity or consciousness, at least without doing the real philosophical work to figure out what that actually means.

> My stance is it's (obviously and only a fool would think otherwise) never going to be conscious because consciousness is a physical process based on particular material interactions, like everything else we've ever encountered.

Seems like you have that backwards. If consciousness is from a nonphysical process, like a soul that's only given to humans, then it follows that you can't build consciousness with physical machines. If it's purely physical, it could be built.

> is a physical process based on particular material interactions,

This is a pretty messy argument as computers have been simulating material interactions for quite some time now.

The problem with adding definitions to words like “thinking” and “free will” is that doing so means humans can no longer pretend they are special.

Even in this thread, the number of people claiming some mystical power separating humans from all the rest of nature is quite noticeable.

You’re mixing and missing a few things here.

1. All previous discussion of thinking was in nature to human and animal minds. The reason this is a question in the first place right now is because we ostensibly have a new thing which looks like a human mind but isn’t. That’s the question at hand here.

2. The question in this particular topic is not about technological “progress” or anything like it. It’s about determining whether machines can think, or if they are doing something else.

3. There are absolutely instances in which the previous word doesn’t quite fit the new development. We don’t say that submarines are swimming like a fish or sailing like a boat. To suggest that “no, actually they are just swimming” is pretty inadequate if you’re trying to actually describe the new phenomenon. AIs and thinking seem like an analogous situation to me. They may be moving through the water just like fish or boats, but there is obviously a new phenomenon happening.

> But the word already means what we think it means.

But that word can mean different things to different people. With no definition, how can you even begin to have a discussion around something?

"No matter what you've read elsewhere, rights aren't given, they're earned. You want rights? Pick up a musket and fight for them, the way we had to."

Even if AI becomes indistinguishable from human output, there will be a fringe group arguing that AI is not technically thinking. Frankly it’s just a silly philosophical argument that changes nothing. Expect this group to get smaller every year.

If the LLM output is more effective than a human at problem solving, which I think we can all agree requires intelligence, how would one describe this? The LLM is just pretending to be more intelligent? At a certain point saying that will just seem incredibly silly. It’s either doing the thing or it’s not, and it’s already doing a lot.

I feel like despite the close analysis you grant to the meanings of formalization and syntactic, you've glossed over some more fundamental definitions that are sort of pivotal to the argument at hand.

> LLMs do not reason. They do not infer. They do not analyze.

(definitions from Oxford Languages)

reason(v): think, understand, and form judgments by a process of logic.

to avoid being circular, I'm willing to write this one off because of the 'think' and 'understand', as those are the root of the question here. However, forming a judgement by a process of logic is precisely what these LLMs do, and we can see that clearly in chain-of-logic LLM processes.

infer(v): deduce or conclude (information) from evidence and reasoning rather than from explicit statements.

Again, we run the risk of circular logic because of the use of 'reason'. An LLM is for sure using evidence to get to conclusions, however.

analyze(v): examine methodically and in detail the constitution or structure of (something, especially information), typically for purposes of explanation and interpretation.

This one I'm willing to go to bat for completely. I have seen LLM do this, precisely according to the definition above.

For those looking for the link to the above definitions - they're the snippets google provides when searching for "SOMETHING definition". They're a non-paywalled version of OED definitions.

Philosophically I would argue that it's impossible to know what these processes look like in the human mind, and so creating an equivalency (positive or negative) is an exercise in futility. We do not know what a human memory looks like, we do not know what a human thought looks like, we only know what the output of these things looks like. So the only real metric we have for an apples-to-apples comparison is the appearance of thought, not the substance of the thing itself.

That said, there are perceptible differences between the output of a human thought and what is produced by an LLM. These differences are shrinking, and there will come a point where we can no longer distinguish machine thinking and human thinking anymore (perhaps it won't be an LLM doing it, but some model of some kind will). I would argue that at that point the difference is academic at best.

Say we figure out how to have these models teach themselves and glean new information from their interactions. Say we also grant them directives to protect themselves and multiply. At what point do we say that the distinction between the image of man and man itself is moot?

> Neither can humans. We also just brute force "autocompletion"

I have to disagree here. When you are tasked with dividing 2 big numbers you most certainly don't "autocomplete" (with the sense of finding the most probable next tokens, which is what an LLM does), rather you go through set of steps you have learned. Same as with the strawberry example, you're not throwing guesses until something statistically likely to be correct sticks.

humans, and even animals track different "variables" or "entities" and distinct things with meaning and logical properties which they then apply some logical system on those properties to compute various outputs. LLMs see everything as one thing, in case of chat-completion models, they're completing text. in case of image generation, they're completing an image.

Look at it this way, two students get 100% on an exam. One learned the probability of which multiple choice options have the likelihood of being most correct based on how the question is worded, they have no understanding of the topics at hand, and they're not performing any sort of topic-specific reasoning. They're just good at guessing the right option. The second student actually understood the topics, reasoned, calculated and that's how they aced the exam.

I recently read about a 3-4 year old that impressed their teacher by reading perfectly a story book like an adult. it turns out, their parent read it to them so much, they can predict based on page turns and timing the exact words that need to be spoken. The child didn't know what an alphabet, word,etc.. was they just got so good at predicting the next sequence.

That's the difference here.

>>> We also just brute force "autocompletion"

Wouldn't be an A.I. discussion without a bizarre, untrue claim that the human brain works identically.

> We also just brute force "autocompletion" with our learned knowledge and combine it to new parts, which we then add to our learned knowledge to deepen the process

you know this because you're a cognitive scientist right? or because this is the consensus in the field?

>Neither can a toddler nor an animal. The level of ability is irrelevant for evaluating its foundation.

Its foundation of rational logical thought that can't process basic math? Even a toddler understands 2 is more than 1.

>But if you add a feedback loop where it can use tools, investigate external files or processes, and then autocomplete on the results, you get to see something that is (close to) thinking

It's still just information retrieval. You're just dividing it into internal information (the compressed representation of the training data) and external information (web search, API calls to systems, etc). There is a lot of hidden knowledge embedded in language and LLMs do a good job of teasing it out that resembles reasoning/thinking but really isn't.

>>you get to see something that is (close to) thinking.

Isn't that still "not thinking"?

Just ask it how many r's are in strawberry and you will realize there isn't a lot of reasoning going on here, it's just trickery on top of token generators.

Sorry to say, Claude Sonnet 4.5 (with and without extended thinking) has this one solved now. So, not “any” LLM, but maybe some still need work.

I presume they finally got around to training the internal reasoning loop into predicting when it needs to apply answer predictions character-by-character.

—- Transcripts from Claude Sonnet 4.5, initially with extended thinking:

> How many a’s in the phrase “artichokes and aardvarks”? Be sure to check your answer.

> Let me count the letter ‘a’ in “artichokes and aardvarks”: • artichokes: 1 ‘a’ (at the beginning) • and: 1 ‘a’ (at the beginning) • aardvarks: 3 ‘a’s (two at the beginning: “aa”, and one in the middle: “ardvarks”) Total: 1 + 1 + 3 = 5 ‘a’s Let me verify by marking them: artichokes and aardvarks Yes, there are 5 letter ‘a’s in the phrase “artichokes and aardvarks”.

> Are you sure that’s correct?

> Let me recount more carefully: “artichokes and aardvarks” Going through each word letter by letter: 1. artichokes - has 1 ‘a’ (first letter) 2. and - has 1 ‘a’ (first letter) 3. aardvarks - has 3 ‘a’s (first letter, second letter, and one more in the middle) Total: 1 + 1 + 3 = 5 ‘a’s Yes, I’m confident the answer is 5. Is there something specific you’d like me to recheck?

> How many r’s in strawberry?

> Let me count the letter ‘r’ in “strawberry”: s-t-r-a-w-b-e-r-r-y Going through each letter: • s: not an r • t: not an r • r: 1st r • a: not an r • w: not an r • b: not an r • e: not an r • r: 2nd r • r: 3rd r • y: not an r There are 3 r’s in “strawberry”.

<Without extended thinking> > How many u’s in universalulis?

> Looking at “universalulis” letter by letter: u-n-i-v-e-r-s-a-l-u-l-i-s There are 2 u’s in “universalulis”.

For what it's worth, when I ask ChatGPT 5, it gets the correct answer every time. The response varies, but the answer is always three.

> This does not preclude reasoning.

It does not imply it either. to claim reasoning you need evidence. it needs to reliably NOT hallucinate results for simple conversations for example (if it has basic reasoning).

> Because I see them solve real debugging problems talking through the impact of code changes or lines all the time to find non-obvious errors with ordering and timing conditions on code they’ve never seen before.

Programming languages and how programs work are extensively and abundantly documented, solutions to problems and how to approach them,etc.. have been documented on the internet extensively. It takes all of that data and it completes the right text by taking the most correct path way based on your input. it does not actually take your code and debug it. it is the sheer volume of data it uses and the computational resources behind it that are making it hard to wrap your head around the difference between guessing and understanding. You too can look at enough stack overflow and (poorly) guess answers for questions without understanding anything about the topic and if you guess enough you'll get some right. LLMs are just optimized to get the amount of correct responses to be high.

And because ALL the marketing AND UX around LLMs is precisely trying to imply that they are thinking. It's not just the challenge of grasping the ridiculous amount of resources poured in, which does including training sets, it's because actual people are PAID to convince everybody those tools are actually thinking. The prompt is a chatbox, the "..." are there like a chat with a human, the "thinking" word is used, the "reasoning" word is used, "hallucination" is used, etc.

All marketing.

> Any thinking that happens with words is fundamentally no different from what LLMs do.

This is such a wildly simplified and naive claim. "Thinking with words" happens inside a brain, not inside a silicon circuit with artificial neurons bolted in place. The brain is plastic, it is never the same from one moment to the next. It does not require structured input, labeled data, or predefined objectives in order to learn "thinking with words." The brain performs continuous, unsupervised learning from chaotic sensory input to do what it does. Its complexity and efficiency are orders of magnitude beyond that of LLM inference. Current models barely scratch the surface of that level of complexity and efficiency.

> Do you have a concept of one-ness, or two-ness, beyond symbolic assignment?

Obviously we do. The human brain's idea of "one-ness" or "two-ness" is grounded in sensory experience — seeing one object, then two, and abstracting the difference. That grounding gives meaning to the symbol, something LLMs don't have.

We do a lot of autocompletion and LLMs overlap with that for sure. I don't know about the "vast majority" even basic operations like making sure we're breathing or have the right hormones prompted are not guesses but deterministic algorithmic ops. Things like object recognition and speech might qualify as autocompletion. But let's say you need to setup health-monitoring for an application. that's not an autocomplete operation. you must evaluate various options, have opinions on it, weigh priorities,etc.. in other words, we do autocompletion but even then the autocompletion is a basic building block or tool we use in constructing more complex decision logic.

If you train an animal to type the right keys on a keyboard that generates a hello world program, you didn't just teach them how to code. they just memorized the right keys that lead to their reward. a human programmer understands the components of the code, the intent and expectations behind it, and can reason about how changes would affect outcomes. the animal just knows how the reward can be obtained most reliably.

I think (hah) you're understimating what goes on when living things (even small animals) think. We use auto-compleition for some tasks, but it is a component of what we do.

Let's say your visual system auto-completes some pattern and detects a snake while you're walking, that part is auto-completion. You will probably react by freezing or panicing, that part is not auto-compleition, it is a deterministic algorithm. But then you process the detected object, auto-compleiting again to identify it as just a long cucumber. But again, the classification part is auto-completion. What will you do next? "Hmm, free cucumber, i can cook with it for a meal" and you pick it up. auto-completion is all over that simple decision, but you're using results of auto-completion to derive association (food), check your hunger level (not auto-completion), determine that the food is desirable and safe to eat (some auto-compleition), evalute what other options you have for food (evaluate auto-complete outputs), and then instruct your nervous system to pick it up.

We use auto-compleition all the time as an input, we don't reason using auto-compleition in other words. You can argue that if all your input is from auto-completion (it isn't) then it makes no difference. But we have deterministic reasoning logical systems that evaluate auto-completion outputs. if your cucumber detection identified it as rotten cucumber, then decision that it is not safe to eat is not done by auto-completion but a reasoning logic that is using auto-completion output. You can approximate the level of rot but once you recognize it as rotten, you make decision based on that information. You're not approximating a decision, you're evaluating a simple logic of: if(safe()){eat();}.

Now amp that up to solving very complex problems. what experiments will you run, what theories will you develop, what R&D is required for a solution,etc.. these too are not auto-completions. an LLM would auto complete these and might arrive at the same conclusion most of the time. but our brains are following algorithms we developed and learned over time where an LLM is just expanding on auto-completion but with a lot more data. In contrast, our brains are not trained on all the knowledge available on the public internet, we retain a tiny miniscule of that. we can arrive at similar conclusions as the LLM because we are reasoning and following algorithms matured and perfected over time.

The big take away should be that, as powerful as LLMs are now, if they could reason like we do, they'd dominate us and become unstoppable. Because their auto-completion is many magnitudes better than ours, if they can write new and original code based on an understanding of problem solving algorithms, that would be gen ai.

We can not just add 1 + 1 but prove that the addition operation is correct mathematically. and understand that when you add to a set one more object, the addition operation always increments. We don't approximate that, we always, every single time , increment because we are following an algorithm instead of choosing the most likely correct answer.

A long time ago I noticed that I sometimes already had a complete thought before my inner monologue turned it into words. A few times I tried skipping the inner monologue because I'd clearly already thought the thought. Turns out the bit of my brain that creates the inner monologue from the thought, can generate a sense of annoyance that the rest of my brain can feel.

Not that it matters, there's evidence that while LLMs output one word at a time, they've got forward-planning going on, having an idea of the end of a sentence before they get there.

Tell that to German-speakers, where the verb comes last, and the order of things in sentences is not anything like English, therefore requiring you to think of the entire sentence before you just spit it out. Even the numbers are backwards (twenty-two is two-and-twenty) which requires thinking.

Furthermore, when you ask an LLM to count how many r's are in the word strawberry, it will give you a random answer, "think" about it, and give you another random answer. And I guarantee you out of 3 attempts, including reasoning, it will flip-flop between right and wrong, but unlike a human, it will be random, because, unlike humans who, when asked "how many r's are in the word strawberry" will not be able to tell you the correct answer every. fucking. time.

edit: formatting

Humans with certain amnestic syndromes are incapable of learning. That doesn't make them unintelligent or incapable of thought.

Most AI tooling is shipped with a feedback loop around the LLM. The quality of Claude Code for example lies in the feedback loop it provides on your code. Maybe the LLM itself isn't thinking, but the Agent which ships an LLM plus feedback loop definitely shows thinking qualities.

Just now in an debugging session with claude code:

  * let me read this file...
  * let me read this file...
  * I think there's a caching issue with the model after dropping the module. Let me check if there's a save or reload needed after DROP MODULE. First, let me verify something:
  * creates a bash/javascript script to verify its assumption
  * runs the script (after review and approval)
  * Aha! I found the problem! Look at the output...

How is this not thinking?

You're moving the goalposts and contradicting yourself with language games.

Something doesn't need to learn to think. I think all the time without learning.

There's also an argument for machines already starting to crack learning with literal reinforcement training and feedback loops.

Your language game was when you said the 'cognition ends...', as cognition is just a synonym for thinking. "The thinking ends when the inference cycle ends. It's not thinking'" becomes a clear contradiction.

As for "the process by which it does that is wholly unrelated", buddy it's modelled on human neuron behaviour. That's how we've had this generative AI breakthrough. We've replicated human mental cognition as closely as we can with current technology and the output bears striking resemblance to our own generative capabilities (thoughts).

Happy to admit it's not identical, but it's damn well inside the definition of thinking, may also cover learning. It may be better to take a second look at human thinking and wonder if it's as cryptic and deep as we thought ten, twenty years ago.

I’d say it’s more like annealing. Sort, filter, compress, integrate.

Updates your models for the next morning, which is why the answer is there when it wasn’t before.

I assume most people agree with that framing. “Thinking” generally implies a level of consciousness or self-awareness.

“Let me think about this.” “I have to think on it.”

My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.

Sometimes I think people leveraging criticisms of LLMs used ChatGPT 3 years ago and haven't touched one since, except for asking how many r's are in strawberry a year and a half ago.

Yeah, as someone who has gained a lot of interaction skills by playing with the constructivist learning ennvironment called the enneagram, I can attest that it much resembles behaviour characteristic of certain enneatypes.

Yes, often

What now, two minutes using one and you are going to get that!

That's not actually true - try photocopying a banknote with the security pattern on it.

So, if there's a flaw in its sensor and you get somewhat warped output, would you consider it thinking then?

It is the wrong category error.

The proper category error in the context of the discussion would be to say the photocopier is drawing a picture.

It doesn't matter how well or not the photocopier recreates an image. To say the photocopier is drawing a picture is just nonsense and has no meaning.

The same category error as to say the LLM is "thinking".

Of course, the category error could be well exploited for marketing purposes if you are in the business of selling photocopiers or language models.

That's not a flaw. That model's creativity tuned a bit too high. It's a happy little copier which can be a little creative and unconventional with reasoning, at times.

The fact that a model trained on the internet, on which the correct rules of chess are written, is unable to determine what is and is not a legal move, seems like a sign that these models are not reasoning about the questions asked of them. They are just giving responses that look like (and often are) correct chess moves.

> Learn to use the tools.

Thing is, you wouldn't need to learn to use the tools if the tool was able to think. A thinking entity is able to adapt to other parties who lack learnings. This confirms that LLMs are little more than fancy RNGs.

> I use it daily and get meaningful, quality results.

That's what the tea leaf readers say too, funnily enough.

Learn to work on interesting problems? If the problem you are working on is novel and hard, the AI will stumble.

Generalizing your experience to everyone else's betrays a lack of imagination.

They are only meaningful and quality if you don’t know what you’re doing. But please do show some of this meaningful and quality work so I can be proven wrong.

That's a skill issue on your end

> You're just assuming that mimicry of a thing is not equivalent to the thing itself.

I'm not assuming that, that's literally the definition of mimicry: to imitate closely.

You might say I'm assuming that it is mimicking and not actually thinking, but there's no evidence it's actually thinking, and we know exactly what is IS doing because we created the code that we used to build the model. They're not thinking, it's doing math, mathematical transformations of data.

But a simulated mind is not a mind. This was already debated years ago with the aid of the Chinese Room thought experiment.

>If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

it wouldn't work probably, brains constantly alter themselves by forming new connections. Learning is inseparable from our intelligence.

> If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

Perhaps this is already known, but I would think there is a high chance that our brains require "write access" to function. That is, the very process of neural activity inherently makes modifications to the underlying structure.

Within a session, yes, it will add each input to the context.

Start a new session with a blank context and it will not have learned anything.

I've also experienced this, to an extent, but on qualitative topics the goodness of an answer - beyond basic requirements like being parseable and then plausible - is difficult to evaluate.

They can certainly produce good-sounding answers, but as to the goodness of the advice they contain, YMMV.

how do you definitely know that?

I never met an addicted person that could be reduced to a simple while(true) print("fail") loop.

I haven't tried in a while but at least previously you could completely flummox Gemini by asking it to come up with some plausible English words with no real known meaning; it just kept giving me rare and funny-sounding actual words and then eventually told me the task is impossible.

Ask ChatGPT about ConLang. It knows. Inventing languages was solved a hundred years ago with Esperanto.

> but it still fits nicely within your request for something that no one ever came up with

YOU came up with it dawg

To borrow a line from Dijkstra, the claim seems a bit like saying that a submarine is swimming.

slavery is slavery does not mean it is AI or human. if slavery is ok then the question is who can own a slave. the answer is coporates like open ai. which is terrible for humanity and the universe

I agree. I'm just pointing out that the meaning of the word "think" already applied to the silicon substrate pre-ai, so just saying it's still applicable isn't that compelling.

But yeah, I am fully willing to believe that a silicon based life form could think and be alive. i just don't think we're there. Yes this thing speaks using a passable imitation of the voices of PhDs and poets, but in a way a simulated annelid is more alive.