Comment by hmokiguess
4 hours ago
Tangential but, are there benchmarks out there on how languages affect latent spaces and performance of these models?
This other day I was looking at that “caveman” skill, and was shocked to see it evolved to become a company, and, in one of its modes, the highest form of compression seems to be “Wenyan” which is Classical Chinese.
Should I get started on learning Chinese?
There are lots of papers on the topic. I think the best summary is "it's complicated". Typically models perform slightly better in English, typically best in either professional English or very rude English. Though this varies by model, not all react well to rude English, and I wouldn't be surprised if Chinese was on the rise
Also, "less tokens" is not always straight forward. I doubt it's a coincidence that the cavemen skill (or now proxy, I guess) has lots of numbers, but not a single benchmark on model performance or actual per-task token savings
For example one paper I remember found that without CoT, just stating your prompt twice increases model performance. With CoT, the same function is served by the CoT restating the important parts of your question. Something about which tokens can affect which other tokens in attention implementations
Very interesting, can you link some of the papers if you don't mind? I'm curious about this space.
I'm finding more and more there seem to be sort of niche prompting skills that are important to be aware of
It's interesting to note that the newer LLMs like deepseek v4 or kimi k3 basically use caveman mode natively for their thinking traces. Lot word dropping when thinking.