Comment by TomGarden
3 days ago
The exciting thing about this is in them optimizing for inference on Macs. This, and optimization for other consumer hardware, is where local AI needs to go
3 days ago
The exciting thing about this is in them optimizing for inference on Macs. This, and optimization for other consumer hardware, is where local AI needs to go
Given the size of Macs (2 or 4 of them) required to run it, they're just paying lip service to that option. Other models, e.g. deepseek flash, are much more practical size. Even that takes a 96 or 128GB ram Mac.
Yeah I'm more excited about the prospect of them explicitly saying it, which hopefully can lead to better places down the line.
I thought DS Flash needed 2bit quant to work on 128GB? I have an m4 max 128 GB and the most reasonable model to run still seems to be qwen's 32B A3B