Comment by Catloafdev
7 hours ago
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
7 hours ago
There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.
Imagine the following conversation:
How was your trip to Los Angeles?
There were no movie stars!
Is the second person's statement "objectively false"?
Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?
You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.
[dead]