Comment by mudkipdev
6 days ago
It simply means the tokenizer's training corpus may have included a massive amount of German literature or accidentally oversampled a web page where that word was frequently repeated. Look up "glitch tokens" to learn more.
No comments yet
Contribute on Hacker News ↗