Comment by ashkankiani
6 days ago
I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
6 days ago
I'm guessing this is an allusion to reinventing something that already exists, but do you mind explaining what that is to me, since I don't know?
It is describing essentially how tokenisation is done for LLMs (Large Language Model → Expansive Dictionary Model).
You mean Quantumnisation ;)
I suspect they're talking about LLM tokens.