Comment by tarruda
1 day ago
Hopefully it will be open weights and have the same architecture and size as the current v4 flash vision, which is probably the best LLM that can be run on 128G devices.
1 day ago
Hopefully it will be open weights and have the same architecture and size as the current v4 flash vision, which is probably the best LLM that can be run on 128G devices.
Interesting, I had assumed it'd be too large to fit. What quant and context size are you running?
IQ3_XXS (~3.2 BPW). For me this is an option because my Mac studio is only used for serving LLMs, so I can afford to dedicate most of its RAM to this. I can run with 256k context and only uses ~117G, with the remaining (up to 125G which I can allocate to VRAM) being used for prompt caching and context checkpoints.
I'm making my own quants, though the Vision-Exp version is outdated and won't work on llama.cpp master branch (I built it before llama added support):
- https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF
- https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-...
For the Vision-exp version, I also ran perplexity + KLD against the original MXFP4. Seems quite OK: https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-...
Thanks, I'll give that a try. I basically have the same use case, only on Strix Halo.
1 reply →
Hey there! I do the same but I use dwarfstar at a 2-bit quant: https://github.com/antirez/ds4
I'm curious if you've tried dwarfstar and decided to move to llama.cpp and 3 bit quants or what made you go that route instead? I've been using ds4 for months now and it's already got support for the new vision model, haven't tried it yet, still on 0731 but it's been very solid for me.
1 reply →