Comment by combyn8tor
16 hours ago
Is the load balancing of the relays out of scope? It doesn't seem to be addressed in the write up unless I missed it.
16 hours ago
Is the load balancing of the relays out of scope? It doesn't seem to be addressed in the write up unless I missed it.
EDIT: Sorry I just noticed this was directed to Cloudflare. They're using the same architecture as Cloudflare Realtime, their WebRTC offering.
`relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited.
The relays nodes currently form a mesh network and gossip origins between themselves. I used to work at Twitch on the CDN team so I'd like to eventually add tiers, but it's overkill with near zero users.
The moq-relay and terraform code is all open source if you're super curious.
Anycast can have serious reliability challenges. It was common at GCP for a small QPS user of anycast to have their Load Balancers nuked in a given pop as it was backed by a single machine. But BGP showed it as still the best route. The major DNS based offerings don't have such issues.
QUIC has support for preferred address, where anycast is used for the QUIC handshake then the connection migrates to a unicast address. It still has issues but it's nice to have sticky established connections and avoid flapping mid connection.
I work for a CDN that does DNS steering. DNS record lifetimes are nonzero and can be surprisingly long. But you do get some very fine control over where data goes if resolvers cooperate.
Home much success have you have with GeoDNS? We've seen it fail when users are using privacy respecting resolvers like 1.1.1.1. It gets the continent right but fails on city/state level.
It works well right now because there's only one edge per continent. But if I had traffic, anycast is definitely the way to go.
I plan to cover more of the internal implementation details at a future date, possibly at a conference this fall..
But I can at least say that we use anycast to route to a network-proximal colo.