Comment by adolph
2 days ago
> The fastest protocols are the TLS/HTTP based ones which stream data.
I think maybe you are referring to QUIC [0]? It'd be interesting to see some userspace clients/servers for QUIC that compete with Aspera's FASP [1] and operate on a point to point basis like scp. Both use UDP to decrease the overhead of TCP.
We've been looking at using QUIC as the transport layer in HPN-SSH. It's more of a pain that you might think because it breaks the SSH authentication paradigm and requires QUIC layer encryption - so a naive implementation would end up encrypting the data twice. I don't want to do that. Mostly what we are thinking about doing is changing the channel multiplexing for bulk data transfers in order to avoid the overhead and buffer issues. If we can rely entirely on TCP for that then we should get even better performance.
Yeah, my naive implementation thought experiment was oriented towards a side channel brokered by the ssh connection using nginx and curl. Something like source opens nginx to share a file and tells sink via ssh to curl the file from source with a particular cert.
However, I observed that curl [0] uses openssl' quic implementation (for one of its experimental implementations). Another backend for curl is Quiche [1] which has client and server components already, has the userspace crypto etc. It's a little confusing to me, but CloudFlare also has a project quiche [2] which is a Rust crate with a CLI to share and consume files.
0. https://curl.se/docs/http3.html
1. https://github.com/google/quiche/tree/main/quiche/quic
2. https://github.com/cloudflare/quiche
Available QUIC implementations are very slow. MsQUIC is one of the fastest and can only reach a meager ~7 Gb/s [1]. Most commercial implementations sit in the 2-4 Gb/s range.
To be fair, that is not really a problem of the protocol, just the implementations. You can comfortably drive 10x that bandwidth with a reasonable design.
[1] https://microsoft.github.io/msquic/
Thank you for linking to those benchmarks. Its interesting that in OpenSSL upload, Windows userspace is 16% faster than Linux.
Actually the fastest ones in my experience are the HTTP/1.x ones. HTTP/2 is generally slower in rclone though I think that is the fault of the stlib not opening more connections. I haven't really tried QUIC
I just think for streaming lots of data quickly HTTP/1.x plus TLS plus TCP has received many more engineering hours of optimization than any other combo.
Maybe this is one of those things where "Worse is Better" [0] given HTTP/1.x will always receive more time/attention/resources than something that might be theoretically superior but never got the resources to fulfill its promise. Cloudflare is probably one of the few organizations outside of Google with an internal economic case to support QUIC. For everyone else there is the option of paying IBM for Aspera using FASP.
0. https://en.wikipedia.org/wiki/Worse_is_better