Comment by tredre3
6 hours ago
I'm curious how your project compares to plain mmap!
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower
hm. in linux you have MAP_POPULATE which forces a prefetch where macos relies on page faults and lazy loading. if MADV_WILLNEED doesn't help, maybe readv to vector read directly or mmap+writev (write to a dummy fd, with an iovec for each page, hopefully resulting in a one-syscall-big-pagefault for mmap.) maybe also experiment with a loop that just reads one byte from each relevant page after mmap but before real computation?
Any idea if madvise helps? Admittedly I have very limited experience and only on Linux
I tried it. madvise didn't make mmap better than pread. I also tested F_RDADVISE, it helped on short decodes, but somehow got worse on longer decodes. Not very clear why, most likely problem somewhere at APFS and it is closed source and not much docs for it
MADV_SEQUENTIAL might help a bit, but not that much. Biggest problem here is throughput-vs-latency.
With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput.
With explicit read()/pread(), kernel & SSD can work with much larger chunks, so it's easier to hit maximum bytes/second throughput.
Plus, with modern CPUs, IO-wait could be efficiently combined with number-crunching. So, if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token.
for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?
Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.
And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.
Ya I'd be interested to see a comparison of using llamacpp with ssd offloading to compare real speeds.