Comment by addaon
7 hours ago
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...
Extremely impressive.
Not sure Data (Star Trek TNG) could read that fast.
“Accessing”
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
The base level M5 doesn't just have more memory than the base level M2.
The memory bandwidth is bumped up by 50%, and the size of the on-die system level cache is bumped up by 50% as well.
I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.
It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing
Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?
IIRC, depends on the SSD size.. larger sizes had 2x the bandwidth, so it depends. That's combined/offset with the M5 improvements even further.
The M5 MBP has 24GB of RAM, more context in RAM perhaps?
The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood
Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache.
Try to leave only a gigabyte or two free, speed likely would drop dramatically.
Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.