Comment by krackers

Oh sure, dogs definitely learn, they have about a 2 year old's level of intellect if I remember correctly? I just meant as example to the train/response thing. I believe the issue is their brains just aren't going to continue "leveling up" the way a human's does as it grows. My assumption is that this is because the AI isn't actually "understanding and thinking", but just acting according to its training. A program following source code, rather than rewriting its own source code, as it were.

Parsing? I don't think dogs "parse" language in a decompositional matter.

Maybe it takes some world modeling to do it as well as they do, but ultimately they are just predicting the next token. These things are not mutually exclusive.

There actually is a clean line between flying and not flying. And that's when the lift generated is greater than the pull of earth's gravity. The fact that it "feels" weightless gradually doesn't change the fact that if lift<weight then the plane is not flying. If lift>weight, plane is flying. There is no "semi flying". If it's already airborne and lift becomes less than weight, then it stops flying and starts gliding.

The lift is an emergent behavior of molecules interacting (mostly) with the wings. But there is a hard clean cutoff between "flying" and "not flying".

Actually, the LLMs made me realize John Searle’s “Chinese room” doesnt make much sense

Because languages have many similar concepts so the operator inside the Chinese room can understand nearly all the concepts without speaking Chinese.

And the LLM can translate to and from any language trivially, the inner layers do the actual understanding of concepts.

Go ask the operator of a Chinese room to do some math they weren't taught in school, and see if the translation guide helps.

The analogy I've used before is a bright first-grader named Johnny. Johnny stumbles across a high school algebra book. Unless Johnny's last name is von Neumann, he isn't going to get anything out of that book. An LLM will.

So much for the Chinese Room.

I disagree, I think it's clear in the article that Chomsky thinks a language also should have a human purpose.

The compression we use in languages to not label impossible adjectives against impossible nouns (green ideas is impossible as ideas don't have colors, we could have a suffix on every noun to mark what can be colored and what cannot) is because we need to transfer these over the air, and quickly, before the lion jumps on the hunter. It's one of the many attributes of "languages in the wild" (Chinese doesn't use "tenses" really, can you imagine the compressive value?), and that's what Chomsky says here:

Proceeding further with normal science, we find that the internal processes and elements of the language cannot be detected by inspection of observed phenomena. Often these elements do not even appear in speech (or writing), though their effects, often subtle, can be detected. That is yet another reason why restriction to observed phenomena, as in LLM approaches, sharply limits understanding of the internal processes that are the core objects of inquiry into the nature of language, its acquisition and use. But that is not relevant if concern for science and understanding have been abandoned in favor of other goals.

Understand what he means: you can read a million text through a machine, it will never infer why we don't label adjective and nouns to prevent confusion and "green ideas". But for us it's painfully obvious, we don't have time when we speak to do all that. And I come from a language when we label every noun with a gender, I can see how stupid and painful it is to grasp for foreigners: it doesn't make any sense. Why do we do it ? Ask ChatGPT, will it tell you that it's because we like how beautiful it all sounds, which is the stupid reason why we do that ?

A “complex” cfg is still a cfg, and, giving credence to Chomsky’s hierarchy, remains computationally less complex than natural, context sensitive, grammars. Even a complex cfg can be parsed by a relatively simple program in ways that context-sensitive grammars cannot.

My understanding is that context sensitive grammars _can_ allow for recursive structures that are beyond cfgs, which is precisely why they sit below csgs in terms of computational complexity.

I don’t agree or disagree that LLMs might be, or are, capable of parsing (i.e., perception in Chomsky’s terms, or, arguably, “understanding” in any sense). But that they can learn the grammar of a “complex cfg” isn’t a convincing argument for the reasons you indicate.

I’ve seen ChatGPT generate bad English and I’ve seen the layer or logic / UI re-render the page as I think there is a simple spell checker that kicks in and tells the api to re-render and recheck.

I don’t believe for one second that LLMs reason, understand, know, anything.

There are plenty of times LLMs fail to generate correct sentences, and plenty of times they fail to generate correct words.

Around the time ChatGPT rolled out web search inside actions, you’d get really funky stuff back and watch other code clearly try to catch the run away.

o3 can be hot garbage if you ask it expand a specific point inside a 3 paragraph memo, the reasoning models perform very, very poorly when they are not summarizing.

There are times where the thing works like magic, other times, asking it to write me a PowerShell script that gets users by first and last name has it inventing commands that flags that don’t exist.

If the model ‘understood’, ‘followed, some sort of structure outside parroting stuff it already knows about it would be easy to spot and guide it via prompts. That is not the case even with the most advanced models today.

It’s clear that LLMs work best at specific small tasks that have a well established pattern defined in a strict language or api.

I’ve broken o3 trying to have it lift working python code, into formal python code, how? The person that wrote the code didn’t exactly code it how a developer would code a program. 140 lines of basic grab some data generate a table broke the AI and it had the ‘informal’ solution in the prompt. So no there is zero chance LMMs do more than predict.

And to be clear, it one shot a whole thing for me last night, using the GitHub/Codex/agent thing in VS code, probably saved me 30 minutes but god forbid you start from a bad / edge / poorly structured thing that doesn’t fit the mould.

Good point, thanks for calling that out. I'm honestly not sure myself! On further reflection, it's probably a matter of degrees?

So for example, a soldier is trained, and then does what it is told. But the soldier also has a deep trough of contextual information and "decision weights" which can change its decisions, often in ways it wasn't trained for. Or perhaps to put it another way: it is capable of operating outside the parameters it was given, "if it feels like it", because the information the soldier processes at any given time may make it not follow its training.

A dog may also disobey an order after being trained, but it has a much smaller range of information it works off of, and fewer things influence its decision-making process. (genetics being a big player in the decision-making process, since they were literally bred to do what we want/defend our interests)

So perhaps a chat AI, a dog, and a soldier, are just degrees along the same spectrum. I remember reading something about how we can get AI to be about as intelligent as a 2-year-old, and that dogs are about that smart. If that's the case (and I don't know that it is; I also don't know if chat AI is actually capable of "disobeying", much less "learning" anything it isn't explicitly trained to learn), then the next question I'd have is, why isn't the AI able to act and think like a dog yet?

If we put an AI in a robot dog body and told it to act like a dog, would it? Or would it only act the way that we tell it dogs act like? Could/would it have emergent dog-like traits and spawn new dog lineages? Because as far as I'm aware, that's not how AI works yet; so to me, that would mean it's not actually doing the things we're talking about above (re: dogs/soldiers)