Comment by kees99

MADV_SEQUENTIAL might help a bit, but not that much. Biggest problem here is throughput-vs-latency.

With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput.

With explicit read()/pread(), kernel & SSD can work with much larger chunks, so it's easier to hit maximum bytes/second throughput.

Plus, with modern CPUs, IO-wait could be efficiently combined with number-crunching. So, if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token.