Comment by internet_points
2 days ago
Ooh, you're saying you need to ask it twice to get the same real-world result. I thought you meant the tokenization was less efficient (as if `Endofunctor` were tokenized `Endo`+`Functor` while `AbstractFactory` was a single token or something like that)
Interesting, most functional languages are quite terse and heavily whitespace, so you would expect raw token counts like that to be lower simply because source documents are on average shorter. Even having to make up for disadvantage things like C++ style operators like &&= all having single tokens from heavy use across a variety of languages but fancy functional operators like <=> being much more rare and in worst cases of undertrained models needing multiple tokens to express.
But yes, terser source documents and low raw token counts could mean lower prediction rates and more prediction attempts needed to get intended results (again, especially if the model was undertrained in that particular language).
This is an interesting hypothesis to test. Depending on the embedding model, such less frequently encountered character sequences in the coding domain could indeed produce less optimal tokens. A dedicated model trained from scratch on Haskell would be optimal but wouldn’t generalize well beyond that, but can serve as a deliberately overfitted baseline.
What I meant was really about the output tokens, on the API pricing level of interaction with an LLM.