Comment by Pragmata
14 hours ago
Cant you run it on a pair of DGX spark? Sure it's expensive now, but at the initial retail price that was what, 8k in hardware? Not exactly unachievable
14 hours ago
Cant you run it on a pair of DGX spark? Sure it's expensive now, but at the initial retail price that was what, 8k in hardware? Not exactly unachievable
No, this is not a small model. I believe 4x Sparks is possible but 4.1 is pretty big for a flash model.
MiMo 2.6 Flash is the most capable model you can currently run on 2x Sparks without resorting to a quantised version of something.
I meant quantized. You can get an EXL3 quantized version thats almost the same performance.
My experience running quantised models is that they look the same at first but are all noticeably worse once the context size gets beyond anything trivial.
1 reply →