Comment by pyridines

It hasn’t been passed and no one cares about it because it’s basically an end goal. No lab can hit it so they can’t juice the crazy Turing benchmark 3000 for marketing.

If someone sat me down today with an LLM and a human and both were trying to prove to me they were human, and I can have conversations of arbitrary length, I’d get it right every time.

I thought the same then. But the funny thing is that today, it has become a lot easier to recognize the frontier models as not human. All the load bearing and not x but y, etc… weird

I'm not exactly that you mean by "new to the system", but it seems to me that that definition makes a calculator intelligent, which I can't agree with.

I agree, but it's clear most people need a definition of intelligence that (1) they qualify for and (2) nothing/no one they don't like qualifies for. And they'll keep redefining intelligence until they satisfy both criteria.

It's all so tiring.

Everyone decides what to think on this issue, then finds out facts to support their idea.

As it stands they are massively useful tools, but for generating usable products they require either A) a lot of expert steering or B) a well defined easily verifiable target and a large compute budget. Most people are using them in mode A with good effect, the progress on math has been done in mode B, which is very promising.

Just a year and a half ago their maximal use was rephrase, summarize, and homework-level tasks.

Five years from now? There be dragons.

"But are they generally intelligent?" What a meaningless question!

The Chinese Room mainly just posits a room that passes the Turing Test, which LLMs do pretty well outside of outright adversarial situations.

Intelligence is compression? What do you mean? Intuitively that doesn't seem right.

>What I don't understand is why LLMs haven't been able to do this yet

LLMs are just trained on what humans have said. Why is it surprising that it's still not possible to reconstruct the intelligence that wrote all that by working backwards? Think of your own work experience. When you look at a piece of code, say, are you always able to discern why the person did what they did, just from the code, with no additional context?

Intelligence is compression

That’s a controversial statement.

I've heard that expression before, but I don't think it can be presented and stated so matter of factly. Where does that put bzip?

Abstraction is compression, and abstraction is definitely a core component of intelligence.

Creating the model takes intelligence, but running it doesn’t. I think the point everybody’s revolving around is that the transformer model is an absurdly inefficient and low-fidelity approximation of a system that acts, observes consequences, and incorporates that feedback going forward.

The issue isn’t really harness vs. no harness. IMO it’s about the lack of an internally generated sense of what to attend to. Yes, the KV cache accumulates state and its “attention” (if you can even call it that) changes with context. We’ve even managed to /kinda/ close the loop with agentic tool calling and ‘memory’ systems, but these just close the loop at the level of behavior rather than disposition. All agentic harnesses do is make an LLM responsive to the consequences of its actions without changing the tendencies by which it determines what to retain or avoid.

The ghost you can’t escape from at this point is the origin of that relevance. Where does the pull toward one thing mattering over another actually come from? If you ran Fable 5 on a Turing machine and rewound the tape to the exact same state with the exact same input (incl. PRNG seed), it would spit out the same output every time.

Everyone’s trying to outrun this problem by training more often or increasing model sizes. But all this does is inform your model, from the outside(!), what constitutes a better state. The thing that’s actually doing the determining remains unchanged. Congratulations, you’ve scaled the transition function and tape of your Turing machine until it requires every watt generated by ERCOT, and it still cannot, for the life of it, tell you why it should give a shit.

A trained model generating output from weights, a seed, and some context effectively has next-state that’s a total function of those three things. Whatever behavior appears as ‘selecting what is relevant’ is, underneath, just a transition rule executing, no matter how sophisticated or creative the output looks. It can be fully accounted for by what was fixed before it started executing. Which means whatever criterion it uses for determining what matters was inherited from a structure that was already in place before it encountered the situation.

No amount of pruning or post-training can fix this. These approaches just replace one externally supplied criterion with another. For a system to be truly adaptable, there would have to be some criterion by which it treats one possible change as preferable to another, and that criterion itself would have to come from... somewhere. You can even change your conception of ‘improvement’ (e.g. parameter count, harnesses, self-modification, hell, even its ability to spit out shitty best-selling romance novels onto Amazon) and you still haven’t explained where the normative distinction comes from. Every layer of this problem has its root in a preference that was supplied from somewhere else.

I genuinely don’t know if this issue bottoms out anywhere, at least for the way we currently build these systems. Perhaps the solution is still computable, maybe? Who knows what that would even look like. But I’m fairly confident that it isn’t a bigger tape. I hope nobody solves this in the near future because, well, I’d like to have a job...

It’s not intelligence if you are the one who gives the correlations to the model in the pre-training. It’s Word2Vec, applied. Model doesn’t learn anything. You embed these correlations and build it from there. It just searches the space.

As my AI professor said in the first lecture: “All AI is advanced search”.

fair reply to an hour video, Scott is just so good to hear his talk is better than I can explain it...

go to 24 minutes and 07 seconds.

it's statistically determining what the next word should be based on all the text it's been trained on. It's not intelligence and he shows what probability it puts on each word that it chooses, but also shows a lot of the other words it was thinking of using. In a later part he shows how it uses words that are not the highest probability (and you question why did it go this route, it's not more correct), but the user never sees this, they see what they think is the correct answer always...

he also shows how context you feed it has a lot to do with what it returns... to the point he can get it to return the capital of France is Marseille, just by typing Marseille a bunch of times before the question. Human intelligence doesn't get confused like that.

And it's not a "hallucination", it's just probability of the next token prediction based on the information it's been trained on and fed, it's not intelligence.