Comment by WorldMaker
2 days ago
Interesting, most functional languages are quite terse and heavily whitespace, so you would expect raw token counts like that to be lower simply because source documents are on average shorter. Even having to make up for disadvantage things like C++ style operators like &&= all having single tokens from heavy use across a variety of languages but fancy functional operators like <=> being much more rare and in worst cases of undertrained models needing multiple tokens to express.
But yes, terser source documents and low raw token counts could mean lower prediction rates and more prediction attempts needed to get intended results (again, especially if the model was undertrained in that particular language).
No comments yet
Contribute on Hacker News ↗