Comment by anon291
19 hours ago
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
19 hours ago
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
Fair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.