Comment by CamperBob2

A language model can generate valid sentences that have never existed before -- not just syntactically or statistically valid sentences in a Markovian sense, but semantically valid ones. How could it possibly do that without an implicit awareness of all aspects of language, including grammar and parts of speech?

That implicit awareness is indistinguishable from knowledge, at least to the extent it yields correct results.

And the fact that it was inferred from examples rather than taught explicitly is indistinguishable from intelligence.