Interesting article but it gave me a bit of a panic attack. Benchmarking (with TPC or otherwise) is NOT the way to determine the correct approach here; that is strictly only to be used for databases (typically RDBMS) effectively “owning” the complete hardware they are running on. An embedded database might be used in that manner if it’s operating as the backend for a pure crud application that performs ~zero server side rendering, parsing, validation, etc and is essentially just an async http-to-SQLite interface. But more likely than not, an embedded db will be used and deployed on machines (not necessarily even servers) serving many a purpose, and need to perform best both within the confines of the resources available to the machine and in relative terms, necessarily making tradeoffs that might sacrifice performance for “value” in terms of CPU or memory usage.
This isn’t just with regards to benchmarking, it’s an essential consideration *any* time you are taking ownership of the cache away from the kernel, which is the only piece in the stack that has viability into the global state and can be trusted to give back memory under pressure to ensure everything plays nice together. It’s not limited to just databases or even just memory, for example FreeBSD has had greater than its fair share of issues that trace back to the ZFS having a separate cache from the kernel (despite the much tighter integration between the two and presence of various mechanisms to address pathological cases). You can also refer to any comparison or benchmark between the use of spinlocks vs mutexes: spin locks consistently perform better in/on (micro)benchmarks but are almost always actually the worse choice in the grand scheme of things because the benchmarks falsely assume complete and uncontended ownership over system resources.
This isn’t even io_uring specific and I’m hardly the first to bring this up in the context of O_DIRECT.
You’ve misunderstood what they’re saying. Not that the benchmark is invalid, but that it’s an incomplete picture if your use case is a mixed set of applications where DB performance is not the only important thing. Hence the comment about memory - O_DIRECT is optimal in terms of DB performance but then the caches are owned by the application and the kernel isn’t free to discard those caches whenever it wants like it can with the page cache. I’m not actually sold on this argument though - I have an embedded DB that with a very tiny block cache goes very very fast, and uses far less RAM than using the page cache because it knows when read ahead is called for vs when it’s not
What would be the correct way to do benchmarking/profiling here? I ask as I've started to run some for Vinyl cache to see what performance issues I can find, and I've quickly learned just how hard it is to get good, repeatable benchmarks that isolate the right thing.
No easy answer! You’d have to have a myriad of different test suites that as closely resemble your real world usage as possible. Or ship it behind a gate and let your users give you real world feedback if you’re at SQLite (not turso) scale - they’ve done that with a lot of ambiguous optimizations.
If you're doing contiguous readahead in userspace, why not just use preadv? It'll limit you to doing readahead up until the next resident page, but at least in my experiments in Marginalia's index, preadv beats io_uring in all cases you can use a single preadv call to do the full read.
Not sure what you mean. Nothing about preadv lets you indicate you only want to read what's already in the page cache. And io_uring and preadv aren't orthogonal - you can give io_uring a preadv op to do the scattered read instead of issuing separate read OPs although I'm not 100% sure how much of a win that is in practice.
Also, I think you misunderstood the blog as it's describing application read-ahead which is what you have to do when using O_DIRECT.
I implement read-ahead in the application by (optionally) preadv:ing a single read into multiple destination buffers in the pool, leaving them unpinned, since as long as you aren't up against the bandwidth limit of the drive, a larger read is generally as fast as multiple smaller one on modern hardware.
I've tried doing this with io_uring as well, but found just eating the preadv syscall cost was faster.
Depends a lot on the memory pressure. If you can be fairly certain the data is (or will be) resident in memory, mmap is basically unbeatable. If you can't (because the data is larger than RAM or there's other stuff competing for RAM), mmap can have gnarly system-wide performance implications[1].
I’m curious why the choice is
between syscalls and, specifically, io_uring with O_DIRECT. AFAIK Turso is like SQLite and supports multiple processes accessing the same database, and I would expect buffering to be a huge win in some workloads. What’s wrong with io_uring without direct? There’s also the middle ground of RWF_DONTCACHE.
You can use io_uring very well with sequential files, where the kernel will do read-ahead for you, possibly helped by you with fadvise.
O_DIRECT is to be used with files that will be accessed in a random order (random meaning that the order is not predictable by the kernel) and with buffer sizes per access great enough that you want to avoid their copying between kernel and application (e.g. at least a few kByte per access).
Whenever O_DIRECT is used, the programmer takes responsibility to implement an adequate form of read-ahead, based on the access pattern that is predicted for the application, and which cannot be guessed by the kernel.
If the programmer did not implement read-ahead, like it was the case before doing the update described in TFA, that was an incompletely written program. One should not enable O_DIRECT without a complete implementation for its requirements, as that can lead only to lower performance than standard I/O.
Interesting article but it gave me a bit of a panic attack. Benchmarking (with TPC or otherwise) is NOT the way to determine the correct approach here; that is strictly only to be used for databases (typically RDBMS) effectively “owning” the complete hardware they are running on. An embedded database might be used in that manner if it’s operating as the backend for a pure crud application that performs ~zero server side rendering, parsing, validation, etc and is essentially just an async http-to-SQLite interface. But more likely than not, an embedded db will be used and deployed on machines (not necessarily even servers) serving many a purpose, and need to perform best both within the confines of the resources available to the machine and in relative terms, necessarily making tradeoffs that might sacrifice performance for “value” in terms of CPU or memory usage.
This isn’t just with regards to benchmarking, it’s an essential consideration *any* time you are taking ownership of the cache away from the kernel, which is the only piece in the stack that has viability into the global state and can be trusted to give back memory under pressure to ensure everything plays nice together. It’s not limited to just databases or even just memory, for example FreeBSD has had greater than its fair share of issues that trace back to the ZFS having a separate cache from the kernel (despite the much tighter integration between the two and presence of various mechanisms to address pathological cases). You can also refer to any comparison or benchmark between the use of spinlocks vs mutexes: spin locks consistently perform better in/on (micro)benchmarks but are almost always actually the worse choice in the grand scheme of things because the benchmarks falsely assume complete and uncontended ownership over system resources.
This isn’t even io_uring specific and I’m hardly the first to bring this up in the context of O_DIRECT.
TPC-H is the recommended approach in Turso's CONTRIBUTING.md - https://github.com/tursodatabase/turso/blob/main/CONTRIBUTIN...
You’ve misunderstood what they’re saying. Not that the benchmark is invalid, but that it’s an incomplete picture if your use case is a mixed set of applications where DB performance is not the only important thing. Hence the comment about memory - O_DIRECT is optimal in terms of DB performance but then the caches are owned by the application and the kernel isn’t free to discard those caches whenever it wants like it can with the page cache. I’m not actually sold on this argument though - I have an embedded DB that with a very tiny block cache goes very very fast, and uses far less RAM than using the page cache because it knows when read ahead is called for vs when it’s not
What would be the correct way to do benchmarking/profiling here? I ask as I've started to run some for Vinyl cache to see what performance issues I can find, and I've quickly learned just how hard it is to get good, repeatable benchmarks that isolate the right thing.
No easy answer! You’d have to have a myriad of different test suites that as closely resemble your real world usage as possible. Or ship it behind a gate and let your users give you real world feedback if you’re at SQLite (not turso) scale - they’ve done that with a lot of ambiguous optimizations.
If you're doing contiguous readahead in userspace, why not just use preadv? It'll limit you to doing readahead up until the next resident page, but at least in my experiments in Marginalia's index, preadv beats io_uring in all cases you can use a single preadv call to do the full read.
Not sure what you mean. Nothing about preadv lets you indicate you only want to read what's already in the page cache. And io_uring and preadv aren't orthogonal - you can give io_uring a preadv op to do the scattered read instead of issuing separate read OPs although I'm not 100% sure how much of a win that is in practice.
Also, I think you misunderstood the blog as it's describing application read-ahead which is what you have to do when using O_DIRECT.
> Nothing about preadv lets you indicate you only want to read what's already in the page cache.
preadv2 + RWF_NOWAIT
1 reply →
So I have a buffer pool with O_DIRECT reads.
I implement read-ahead in the application by (optionally) preadv:ing a single read into multiple destination buffers in the pool, leaving them unpinned, since as long as you aren't up against the bandwidth limit of the drive, a larger read is generally as fast as multiple smaller one on modern hardware.
I've tried doing this with io_uring as well, but found just eating the preadv syscall cost was faster.
14 replies →
I found out that using mmap and just telling uring to read form there to beat anything else.
Depends a lot on the memory pressure. If you can be fairly certain the data is (or will be) resident in memory, mmap is basically unbeatable. If you can't (because the data is larger than RAM or there's other stuff competing for RAM), mmap can have gnarly system-wide performance implications[1].
[1] Mandatory mmap=poop-emoji link: https://db.cs.cmu.edu/mmap-cidr2022/
1 reply →
Why would you use uring to read from an mmap? Couldn't you just memcpy?
12 replies →
Databases vendors usually find that read outperforms mmap. Mmap being fast is a myth.
3 replies →
I’m curious why the choice is between syscalls and, specifically, io_uring with O_DIRECT. AFAIK Turso is like SQLite and supports multiple processes accessing the same database, and I would expect buffering to be a huge win in some workloads. What’s wrong with io_uring without direct? There’s also the middle ground of RWF_DONTCACHE.
You can use io_uring very well with sequential files, where the kernel will do read-ahead for you, possibly helped by you with fadvise.
O_DIRECT is to be used with files that will be accessed in a random order (random meaning that the order is not predictable by the kernel) and with buffer sizes per access great enough that you want to avoid their copying between kernel and application (e.g. at least a few kByte per access).
Whenever O_DIRECT is used, the programmer takes responsibility to implement an adequate form of read-ahead, based on the access pattern that is predicted for the application, and which cannot be guessed by the kernel.
If the programmer did not implement read-ahead, like it was the case before doing the update described in TFA, that was an incompletely written program. One should not enable O_DIRECT without a complete implementation for its requirements, as that can lead only to lower performance than standard I/O.
This has scrolled past in my feed and every time I've read it as "Io_uring without Radiohead", and I mean you could but what would be the point?
[flagged]
[dead]
[dead]
[flagged]