Comment by npilk

>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.

Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.

If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:

https://www.anthropic.com/research/natural-language-autoenco...

No, they really do one at a time. You're incorrect on that.

Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.

Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.

The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.

In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.

You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.

But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?

everyone in this thread is saying something kind of imprecise and reductive and varies between models and even modifications among the models

There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

Yes, models can reason and plan, which helps them write more coherently. But when they write the final output, it’s still a single generation. It would be like letting a human make notes and write an outline, but not let them use the backspace once they start typing their response.

Presumably you could use the same reasoning trace, run multiple generations, and get different outputs (if the temperature is >0).

But now I’m interested in playing more with Cowork or Claude Code/Codex for prose writing to see if the set of tools there affects outputs at all. I guess you might need a more custom “writing” harness.

Isn't this just chain-of-thought though, doing the same thing multiple times without necessarily defining one path?

Sure, but mightn’t there be several plausible long horizon plans?

Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was:

—

*Long journey tracks*

Clinic — “The Return of Evil Bill”

Guided by Voices — not really, wrong band

Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic.

—

So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact.

Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time.

Human writing is like 90% editing.

It can but it is limited because it's only got a single pass through the network to fit the entire "longer horizon plan".

After "stoned" you triggered into a different state.

You be a human who's brain shifted into LLM mode (chainneling Markov?).

Or perhaps you're an LLM impersonating humanity.

I often wonder how much LLMs are just mirroring our own brain's patterns.

You're thinking of Markov chains.

Today I learned a new word, "Oulipo". Interesting.

But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?

I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.