Comment by kaoD
2 months ago
> the LLM perceives [...] the word as a single indivisible thing (a token).
Two actually, "blue" and "berry". https://platform.openai.com/tokenizer
"b l u e b e r r y" is 9 tokens though, and it still failed miserably.
2 months ago
> the LLM perceives [...] the word as a single indivisible thing (a token).
Two actually, "blue" and "berry". https://platform.openai.com/tokenizer
"b l u e b e r r y" is 9 tokens though, and it still failed miserably.
No comments yet
Contribute on Hacker News ↗