Comment by bitexploder
16 hours ago
QFN Q4 K XL Unsloth is basically identical to the bf16 model. It’s KL divergence is very very small. It is competitive with the full model. Also I benchmarked iq3 xxs on Strata for 3 days against deepswe and it scored better than the published benchnarks for the model. If the gsq rco quant prettain tasks fit well it is a great model quant. My data shows it takes 20% more tokens basically than q4 xl. A4 xl is within noise on all my repeated task benchmarks over the public model serving and my own runs of q8. I published the deeswe results. You have to be very specific about model choice and know these models and their quirks.
This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
https://stealthgoat.com/fnfl/pelicans.html
Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
> QFN Q4 K XL Unsloth is basically identical to the bf16 model.
Yes, but like original poster I have 16GB of VRAM and 64GB of RAM, I was referring to non-Strata benchmarks :)
I did quickly run a couple things on Strata but haven't gotten around yet to benchmark bigger models/customize them with Strata, but will do rather soon. So far Strata/Swift IQ2_XS performed really well compared to non-Strata/Qwen3.8 quantized variants.
> This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
Yep, totally get it, and to be honest getting to run ablated models sometimes can also be useful.
> Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
My rig is... ~1.5k, maybe 3k if we count the CPU and GPU upgrade since 2019 :-D I really wanted to grab one of the latest mini-PCs (Halo, Spark, and other minor assemblers) for local setup, but to be honest I can't justify a 6k+ purchase for me and my wife alone. Perhaps in 3 years these mini-PCs will get to the ~2k ballpark (perhaps used), and although older by then, I might be able to run things 24/7 locally as well.
With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.
Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)