Comment by zzzcpan
6 years ago
Say you have a few nodes behind a few different ISPs sharded to clients. Once one node becomes unavailable it gets replaced by another node and back when it becomes available again. This means either all nodes at once can get attacked, but with lower volume or one by one, but affecting only one shard of users for a short period of time it takes to failover.
But in practice datacenters, uplinks and internet exchanges often are able to do flowspec, firewall rules, block all UDP for a subnet in all networks they have relationships with, etc. So plenty of those nodes can be behind ISPs that mitigate volumetric attacks automatically, so even simple DNS failover might be good enough to protect from such attacks. It's not that hard. Layer 7 is where the hard part is.
For reference, having been in this situation before:
1. Having had ~25 servers per datacenter
2. 5 Datacenters (1 in Texas, 1 in Utah, 2 in California, 1 in Chicago)
3. 1 Server in each location connected 10gbit, the rest 1 gbit.
I got to watch first hand as DNS reflection attacks crippled our infrastructure one server at a time. Only 2 of the datacenters (1 in LA, 1 in Chicago) had the infrastructure to mitigate the DDoS without significantly effecting their operations. Even post mitigation, the 2 datacenters that didn't end-up blackhole-ing our IPs at the edge still let so much malicious traffic through that only the 2 10Gbit servers remained online and they were nearly CPU bound over 24 cores just handling all the SENDQ/RECVQ for the NIC.
I mention this because it's sometimes easy to dismiss until you're in the situation and the realities of what you have control over are vastly different from technically feasible. The size and scope of modern DDoS attacks can easily overwhelm entire uplinks to a datacenter, even after pushing mitigations upstream. The reason these reverse proxies from companies like Cloudflare have become so popular is because most will not have the raw resources required to mitigate this themselves. Even some larger datacenters don't have the resources.
> I mention this because it's sometimes easy to dismiss until you're in the situation and the realities of what you have control over are vastly different from technically feasible.
I understand, but you are still talking about a situation where surviving a volumetric DDoS attack without a global centralized provider was possible. It wasn't smooth for you, but it could have been if things were done a bit differently.
Anyway, here on the other side of the world it's not like that, DDoS protection is more common. Because in the early days of DDoS attacks with all the dreadful blackholing one of the big European providers OVH invested in DDoS protection and kind of pushed the whole market to provide it too instead of blackholing.
This is true, it's certainly gotten better since OVH introduced it. It's not so niche as BlackLotus, GigeNET, Arbor Networks, etc...