Comment by skrebbel
5 years ago
In what way is LMDB better than eg SQLite or Redis? For what kinds of use cases would you recommend it?
5 years ago
In what way is LMDB better than eg SQLite or Redis? For what kinds of use cases would you recommend it?
LMDB and SQLite are not directly comparable. LMDB is a transactional B+tree-based key/value store. SQLite is an implementation of a transactional SQL data model on top of a B+tree-based key/value store, so it is logically at least one abstraction layer higher than LMDB. (Key/value stores underlie pretty much all of the other data models you'll ever use.)
That aside - LMDB is not just smaller, faster, and more reliable than SQLite, it is also smaller/faster/more reliable than SQLite's own B+tree implementation, and SQLite can be patched to use LMDB instead of its own B+tree code, resulting in a smaller/faster footprint for SQLite itself.
Proof of concept was done here https://github.com/LMDB/sqlightning
A new team has picked this up and carried it forward https://github.com/LumoSQL/LumoSQL
Generally, unless your application has fairly simple data storage needs, it's better to use some other data model built on top of LMDB than to use it (or any K/V store) directly. (But if building data storage servers and implementing higher level data models is your thing, then you'd most likely be building directly on top of LMDB.)
It's very simple. Single C file implementation. Binary blob keys : binary blob values. The end. If all you need is a bunch of blobs written and read back reliably while in a statistically unreliable situation, LMDB is great.
Well, sqlite is also just a single .c with a single .h.
Sqlite is dozens of files. See https://www3.sqlite.org/cgi/src/dir?ci=tip&name=src
1 reply →
For my applications: latency. The fact that writers do not block readers is just a nice bonus.
Do you reckon LMDB would be reasonably performant compared to raw mmaped files for zero-copy passing large images between processes? I want something like a ring buffer but mitigate the risk of use-after-free if the consumer lags. Seems like lmdb automatically re-uses memory that's long been freed but is sensible to detect that an incoming write needs more space.
Python/c++.
I've looked into mmap and flock but it's messy and not highly portable.
LMDB is in fact not much more than large mmaped file and clever synchronization mechanism on top of that that also serves as dictionary implementation and transaction mechanism. My uses of LMDB are essentially replacements for mmaped file and LMDB gives me the ability to sanely do partial updates.
1 reply →