Comment by Roxxik

My recommendation if you are willing to push it further is to keep KV and always needed weights pinned in memory, so they do not get evicted. This is likely already the case, as they are touched on each token. mmap is slow on evicted pages, as it does not load the whole tensor, but only the touched pages. And it does this through a page fault, thus blocking your code. So it loads a page hands control back to you and the code goes on to touch another evicted page, repeating the loop. Now on an HDD that is not a big problem (yes reads can be coalesced, but a HDD is fundamentally serial in reading) while an SSD can overlay reads better and it is good to keep a few reads in flight at all times to keep its queue fed.

One option is locking the pages. But for that size you need extra privileges.

I experimented with some options. For example: one problem with io_uring is that it still reads to page cache so your reads gets copied in memory after they landed. Now if you pass O_DIRECT that does not happen, but it has its own can of worms.

For full transparency: I had opus write the io_uring layer into llama.cpp for me. And it yielded something slightly short of a 2x tok/s speedup vs simple mmaping. Also I noticed that disabling the warmup and initial test dramatically increases startup time.

The medusa paper looks interesting. My work was a few months ago, multi-token decoding was not a thing then.