Comment by uroni
4 days ago
That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.
Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.
Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever
LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?
The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.