Comment by Havoc
3 days ago
Is there something different about that accuracy measure? i.e. relative to perplexity term usually used
Going from BF16 to 2.8 and losing only ~5% sounds odd to me.
3 days ago
Is there something different about that accuracy measure? i.e. relative to perplexity term usually used
Going from BF16 to 2.8 and losing only ~5% sounds odd to me.
It's accuracy across GSM8K, MMLU, IFEVAL and LiveCodeBench.
They detail their methodology here: https://byteshape.com/blogs/Qwen3-4B-I-2507/