Why? Sqlite and LMDB make fantastic use of it. For anyone doing a single writer db it's a no brainer. It does so much for you and it does it very well. All the things you don't have to implement because it does it for you:
- Reading the data from disk
- Concurrency between different threads reading the same data
- Caching and buffer management
- Eviction of pages from memory
- Playing nice with other processes in the machine
Why would you not leverage it? It's such a great fit for scaling reads.
The strongest argument as far as I can see it is... the problem is you now lose control over all those things. It's a black box with effectively no knobs.
Anyways, read for yourself, Pavlo & Leis get into it in detail, and there's benchmarks:
Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning
Andy is very critical of using mmap in database implementations.
Andy's critiques are only valid on dedicated database servers.
https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...
LMDB uses mmap and Andy recommends LMDB, in the very article this thread is about.
Why? Sqlite and LMDB make fantastic use of it. For anyone doing a single writer db it's a no brainer. It does so much for you and it does it very well. All the things you don't have to implement because it does it for you:
- Reading the data from disk
- Concurrency between different threads reading the same data
- Caching and buffer management
- Eviction of pages from memory
- Playing nice with other processes in the machine
Why would you not leverage it? It's such a great fit for scaling reads.
The strongest argument as far as I can see it is... the problem is you now lose control over all those things. It's a black box with effectively no knobs.
Anyways, read for yourself, Pavlo & Leis get into it in detail, and there's benchmarks:
https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf
https://db.cs.cmu.edu/mmap-cidr2022/
1 reply →
Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning
1 reply →
“ It's such a great fit for scaling reads.”
And losing them.
1 reply →