Comment by observationist

And airplanes don't really fly. And submarines don't swim. And there aren't any real horses powering your engine.

The difference, of course, is that intelligence is the thing that is done, a subset of computation, and all computation is substrate independent. You can definitely argue that LLMs are less intelligent than humans. This is obvious, for the time being, and easy to demonstrate. Saying they are not intelligent is simply untrue.

Whether you want to go to a formalized definition of intelligence, like AIXI, or a neuroscience definition, or the vernacular definition, LLMs are intelligent. The idea that they're random number stochastic functions that are just partially producing sensible results is about 6 years past its expiration date, and if you've been holding on to that idea, it's really time to update your model.

ALICE bots or ELIZA back in the day had a "sense" of intelligence. Modern LLMs are more intelligent than the average human.