Comment by plqbfbv
18 hours ago
I've benchmarked quite a few models on my 9070XT, and I basically gave up hope of using models locally: https://claude.ai/artifact/77pHgucqQNsmic8ZEnqdBA
I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.
I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.
Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.
Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.
QFN Q4 K XL Unsloth is basically identical to the bf16 model. It’s KL divergence is very very small. It is competitive with the full model. Also I benchmarked iq3 xxs on Strata for 3 days against deepswe and it scored better than the published benchnarks for the model. If the gsq rco quant prettain tasks fit well it is a great model quant. My data shows it takes 20% more tokens basically than q4 xl. A4 xl is within noise on all my repeated task benchmarks over the public model serving and my own runs of q8. I published the deeswe results. You have to be very specific about model choice and know these models and their quirks.
This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
https://stealthgoat.com/fnfl/pelicans.html
Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
> QFN Q4 K XL Unsloth is basically identical to the bf16 model.
Yes, but like original poster I have 16GB of VRAM and 64GB of RAM, I was referring to non-Strata benchmarks :)
I did quickly run a couple things on Strata but haven't gotten around yet to benchmark bigger models/customize them with Strata, but will do rather soon. So far Strata/Swift IQ2_XS performed really well compared to non-Strata/Qwen3.8 quantized variants.
> This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
Yep, totally get it, and to be honest getting to run ablated models sometimes can also be useful.
> Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
My rig is... ~1.5k, maybe 3k if we count the CPU and GPU upgrade since 2019 :-D I really wanted to grab one of the latest mini-PCs (Halo, Spark, and other minor assemblers) for local setup, but to be honest I can't justify a 6k+ purchase for me and my wife alone. Perhaps in 3 years these mini-PCs will get to the ~2k ballpark (perhaps used), and although older by then, I might be able to run things 24/7 locally as well.
With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.
Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)