Comment by quotemstr
4 days ago
You're right about that.
Linux needs a way for userspace processes to participate in the kernel's shrinker system for reclaiming memory under pressure. Watching memory PSI is too coarse. MADV_FREE is too complicated and indiscriminate. You could imagine a notification FD, but then you've just reinvented PSI. You could imagine a synchronous signal, but everyone hates signals and won't couple any new functionality to them.
Shrinker-BPF attached to a memfd perhaps? A BPF shrinker could not only choose which pages to evict in a non-stupid way, but could notify userspace in some sane manner (e.g. setting a bitmask somewhere) that it's done so.
(Zero-fill as "notification" is insane and doesn't actually work because zero is a perfectly valid value in a lot of contexts.)
That’s why you do your own memory accounting in the database. Of course, that assumes you own the machine; for an “embedded” DB like LMDB something like PSI may be necessary.
Another possibility for reclaiming physical memory beside unused page decommit with MADV_FREE/MADV_DONTNEED (there might not be unused pages to decommit) is to manually page out cold anonymous pages with MADV_COLD/MADV_PAGEOUT (thanks Android). You can combine this with low swappiness so anonymous pages are unlikely to be paged out automatically when there are clean file-backed pages that can be reclaimed.