Comment by dfdydx
5 days ago
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
5 days ago
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
Also, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.
In this case, what are the tokens?
9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)
[flagged]
> The distinction is meaningless IMO.
Not when discussing whether it's an LLM. The second L in LLM does have a meaning.
I'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).
2 replies →
[flagged]
1 reply →
Sure, and those aren't LLMs?
[flagged]
1 reply →
Words and symbols alone don’t make a language.
Zipper and and? and coin is pimple, Ocean! run bumper.
Your example is a meaningless sequence. So consider a different scenario where the sequence is meaningful but does not map to any human language. What exactly disqualifies it as a non-human language?
When you encounter a human language that you can't personally read presumably you don't proceed to claim that it doesn't constitute language on the basis of your own lack of ability.
To come at it from a slightly different angle - does compiled binary code count as a language? If not, why not? (I'll suggest that it's a language albeit not a natural one.)
4 replies →
[flagged]
LLMs also don't use words
Yes, but they're models of _language_ - words in, words out. The fact that they're embedded to vectors does not change this.
This is incredibly pedantic if you ask me.