Comment by qwertox
2 days ago
I recently had to add `ssl_preread_server_name` to my NGINX configuration in order to `proxy_pass` requests for certain domains to another NGINX instance. In this setup, the first instance simply forwards the raw TLS stream (with `proxy_protocol` prepended), while the second instance handles the actual TLS termination.
This approach works well when implementing a failover mechanism: if the default path to a server goes down, you can update DNS A records to point to a fallback machine running NGINX. That fallback instance can then route requests for specific domains to the original backend over an alternate path without needing to replicate the full TLS configuration locally.
However, this method won't work with HTTP/3. Since HTTP/3 uses QUIC over UDP and encrypts the SNI during the handshake, `ssl_preread_server_name` can no longer be used to route based on domain name.
What alternatives exist to support this kind of SNI-based routing with HTTP/3? Is the recommended solution to continue using HTTP/1.1 or HTTP/2 over TLS for setups requiring this behavior?
Clients supporting QUIC usually also support HTTPS DNS records, so you can use a lower priority record as a failover, letting the client potentially take care of it. (See for example: host -t https dgl.cx.)
That's the theory anyway. You can't always rely on clients to do that (see how much of the HTTPS record Chromium actually supports[1]), but in general if QUIC fails for any reason clients will transparently fallback, as well as respecting the Alt-Svc[2] header. If this is a planned failover you could stop sending a Alt-Svc record and wait for the alternative to timeout, although it isn't strictly necessary.
If you do really want to route QUIC however, one nice property is the SNI is always in the first packet, so you can route flows by inspecting the first packet. See cloudflare's udpgrm[3] (this on its own isn't enough to proxy to another machine, but the building block is there).
Without Encrypted Client Hello (ECH) the client hello (including SNI) is encrypted with a known key (this is to stop middleboxes which don't know about the version of QUIC breaking it), so it is possible to decrypt it, see the code in udpgrm[4]. With ECH the "router" would need to have a key to decrypt the ECH, which it can then decrypt inline and make a decision on (this is different to the TLS key and can also use fallback HTTPS records to use a different key than the non-fallback route, although whether browsers currently support that is a different issue, but it is possible in the protocol). This is similar to how fallback with ECH could be supported with HTTP/2 and a TCP connection.
[1]: https://issues.chromium.org/issues/40257146
[2]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
[3]: https://blog.cloudflare.com/quic-restarts-slow-problems-udpg...
[4]: https://github.com/cloudflare/udpgrm/blob/main/ebpf/ebpf_qui...
> for setups requiring this behavior?
TLS terminating at your edge (which is presumably where the IP addresses attach) isn't any particular risk in a world of letsencrypt where an attacker (who gained access to that box) could simply request a new SSL certificate, so you might as well do it yourself and move on with life.
Also: I've been unable to reproduce performance and reliability claims of quic. I keep trying a couple times a year to see if anything's gotten better, but I mostly leave it disabled for monetary reasons.
> This approach works well when implementing a failover mechanism: if the default path to a server goes down...
I'm not sure I agree: DNS can take minutes for updates to be reflected, and dumb clients (like web browsers) don't failover.
So I use an onerror handler to load the second path. When my ad tracking that looks something like this:
but with the more complex APIs, fetch() is wrapped up similarly in the APIs I deliver to users. This works much better than anything else I've tried.
> […] isn't any particular risk in a world of letsencrypt where an attacker (who gained access to that box) could simply request a new SSL certificate
You can use CAA records with validationmethods and accounturi to limit issuance, so simply access to the machine isn’t enough. (E.g. using dns and an account stored on a different machine.)
For a failover circumstance, I wouldn’t bother with failover for QUIC at all. If a browser can’t make a QUIC connection (even if advertised in DNS), it will try HTTP1/2 over TLS. Then you can use the same fallback mechanism you would if it wasn’t in the picture.
Unfortunately I think that falls under the "Not a bug" category of bugs. Keeping the endpoint concealed all the way to the TLS endpoint is a feature* of HTTP/3.
* I do actually consider it a feature, but do acknowledge https://xkcd.com/1172/
PS. HAProxy can proxy raw TLS, but can't direct based on hostname. Cloudflare tunnel I think has some special sauce that can proxy on hostname without terminating TLS but requires using them as your DNS provider.
Unless you're using ECH (encrypted client helo) the endpoint is obscured (known keys), not concealed.
PS: HAProxy definitely can do this too, something using req.ssl_sni like this:
Then tcp-https-example-proxy is a backend which forwards to a server listening for HTTPS (and using send-proxy-v2, so the client IP is kept). Cloudflare really isn't doing anything special here; there are also other tools like sniproxy[1] which can intercept based on SNI (a common thing commerical proxies do for filtering reasons).
[1]: https://github.com/ameshkov/sniproxy
Neat! Thank you very much for the information.
Hm, that’s a good question. I suppose the same would apply to TCP+TLS with Encrypted Client Hello as well, right? Presumably the answer would be the same/similar between the two.
Not an expert on eSNI, but my understanding was that the encryption in eSNI is entirely separate from the "main" encryption in TLS, and the eSNI keys have to be the same for every domain served from the same IP address or machine.
Otherwise, the TLS handshake would run into the same chicken/egg problem that you have: To derive the keys, it needs the certificate, but to select the certificate, it needs the domain name.
So you only need to replicate the eSNI key, not the entire cert store.
Personally, I'd like to have an option of the outbound firewall doing the eSNI encryption, is that possible?
That fallback instance can then route requests for specific domains to the original backend over an alternate path without needing to replicate the full TLS configuration locally.
Won't you need to "replicate the TLS config" on the back end servers then? And how hard is it to configure TLS on the nginx side anyway, can't you just use ACME?
QUIC v1 does encrypt the SNI in the client hello, but the keys are derived from a predefined salt and the destination connection id. I don't see why decrypting this would be difficult for a nginx plugin.
There is no way to demultiplex incoming QUIC or HTTP/3 connections based on plaintext metadata inside the protocol. The designers went one step too far in their fight against middleboxes of all sorts. Unless you can assign a each destination at least its own (IP address, UDP port) pair you're shit out of luck and can't have end-to-end encryption. A QUIC proxy has to decrypt, inspect, and reencrypt the traffic. Such a great performance and security improvement :-(. With IPv6 you can use unique IP addresses which immediately undoes any of the supposed privacy advantages of encrypting the server name in the first place. With IPv4 your pretty much fucked. Too bad SRV record support for HTTP(S) was never accepted because it would threatten business models. I guess your best bet is to try to redirect clients to unique ports.
Hiding SNI is more important than breaking rare cases of weird web server setups. This setup is not typical because large organizations like Google tend to put all the services behind the same domain name.