Comment by kristopolous

3 months ago

A habana just for inference? Are you sure?

Also I see the 4 bit quants put it at a h100 which is fine ... I've got those at work. Maybe there will be distilled for running at home