Comment by jamesblonde
5 hours ago
I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
You're just going to have to try it. It won't take that long to quantize. Unless the server is currently serving traffic you should be able to get it going in under a day with a frontier model doing the driving for everything - quant, implementation, etc.
One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.
4 bit quants can be fine, try it and see.
4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...
use MoE offloading
> we have restrictions that we can't use a chinese model!
Please expand on this if you can. Why can't you? Who mandated this?
Last I heard, US defense contractors had been directed to not use Deepseek in their operations late last year[0], and most have taken that as an indicator that they shouldn’t be using chinese models generally.
[0] https://digitalpolicyalert.org/change/20659-physical-and-cyb...
US politics. Lot of US corps now ban Chinese models, don’t ask me why, the people in charge are just dumb fucks
Seems like a reasonable policy to me, depending on the sector at the least? Even with open weight / self-hosted models, there's no way to know if the model has been fine-tuned to contain malicious behaviour, triggered only within very specific conditions.
In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.
1 reply →
Executives live in a weird bubble driven largely by consulting firms. Right now those firms are spreading FUD about Chinese models.
2 replies →