Comment by RantyDave
14 hours ago
There was a brief fascination with user mode TCP over DPDK (or similar). What happened with that? Can you get similar performance with QUIC? Does io_uring make it all a moot point?
14 hours ago
There was a brief fascination with user mode TCP over DPDK (or similar). What happened with that? Can you get similar performance with QUIC? Does io_uring make it all a moot point?
I've only done a little prototyping with it, but io_uring addresses the same issue as DPDK, but in a totally different way. If you want high perf, you want to avoid context switches between userland and kernelland; you have DPDK which brings the NIC buffers into userland and bypasses the kernel, you have things like sendfile and kTLS which lets the kernel do most of the work and bypasses userland; and you have io_uring which lets you do the same syscalls as you're doing now, but a) in a batch format, b) also in a continuous form with a submission queue thing. I think it's easier to reach for io_uring than DPDK, but it might not get you as far as DPDK; you're still communicating between kernel and userland, but it's better than normal syscalls.
> Can you get similar performance with QUIC?
I don't know that I've seen benchmarks, but I'd be surprised if you can get similar performance with QUIC. TCP has decades of optimization that you can lean on, UDP for bulk transfer really doesn't. For a lot of applications, server performance from QUIC vs TCP+TLS isn't a big deal, because you'll spend much more server performance on computing what to send than on sending it... For static file serving, I'd be surprised if QUIC is actually competitive, but it still might not be a big deal if your server is overpowered and can hit the NIC limits with either.
It is fairly straightforward to implement QUIC transport at ~100 Gb/s per core without encryption which is comparable or better than TCP. With encryption, every protocol will bottleneck on the encryption and only get a mere 40-50 Gb/s per core unless you have dedicated crypto offload hardware.
However, the highest performance public QUIC implementation benchmarks only get ~10 Gb/s per core. It is unclear to me if this is due to slow QUIC implementations or poor UDP stacks with inadequate buffering and processing.
At least to me, one of the most compelling parts of QUIC is that you establish a connection with TLS without needing extra round trips compared to TCP, where there are separate handshakes for the connection and then the TLS initialization. Even if it was no faster than TCP from that point forward, that seems like enough to make the protocol worthwhile in today's world where TLS is the basically the rule with relatively few exceptions rather than an occasion use case.
It's also something I just find fascinating because it's one of the few practical cases where I feel like the compositional approach has what seems to be an insurmountable disadvantage compared to making a single thing more complex. Maybe there are a lot more of them that just aren't obvious to me because the "larger" thing is already so well-established that I wouldn't consider breaking it into smaller pieces because of the inherent advantage from having them combined, but even then it still seems surprising that that gold standard for so long arguably because of how well it worked with things that came after eventually run into change in expectations that it can't adapt to as well as something with intentionally larger scope to include one of those compositional layers.
If someone with leverage (probably Apple) was willing to put the effort to push it, we could have TCP Fast Open, and you wouldn't need an extra round trip for TCP+TLS. But also note, TLS 1.3 (and TLS 1.2 FalseStart) only add one round trip ontop of TCP; going down from 2 round trips to 1 is nice, but sometimes the QUIC sales sheets claim 3 to 1; if you can deploy QUIC, you can deploy 2 handshake tcp+tls.
Apple put in effort to get MPTCP accepted in cellular networks (where they have direct leverage) and having it out there (used by Siri) puts pressure on other networks too. If they did the same thing for Fast Open (SYN with data), it could be big.
Unfortunately, I'm not sure anyone other than Apple is capable of doing it. Nobody else really has leverage against enough carriers to demand they make new TCP patterns work; and not many organizations would want to try adding something to SYNs that might fail. (Also, MPTCP allows session movement, so a new TLS handshake isn't required)
That is because providing a reliable stream over a stateful connection is actually about a half-dozen layers of abstraction.
TCP couples them all in a large monolithic, tangled mess. QUIC, despite being a little more complex, has the layers much less coupled even though it is still a monolithic blob.
A better network protocol design would be actually fully decoupling the layers then building something like QUIC as a composition of those layers. This is high performance and lets you flexibly handle basically the entire gamut of network protocols currently in use.