Comment by jedberg
7 years ago
The irony of all of this is that those TTLs are almost meaningless as a server operator anyway. Even if you set your TTL to 5 minutes, there are a whole lot of clients that will ignore it.
When I made a DNS switch at reddit, even with a 5 minute TTL, it still took an hour for 80% of the traffic to shift. After a week, only 95% had shifted. After two weeks we still had 1% of traffic going to the old IP.
And after a month there was still some traffic at the old endpoint. At some point I just shut off the old endpoint with active traffic (mostly scrapers with hard coded IPs at that point as far as I could tell).
One of my friends who ran an ISP in Alaska told me that they would ignore all TTLs and set them all to 7 days because they didn't have enough bandwidth to make all the DNS queries to the lower 48.
So yeah, set your TTL to 40 hours. It won't matter anyway. In an emergency, you'll need something other than DNS to rapidly shift your traffic (like a routed IP where you can change the router configs).
It was some time ago, but I've had similar trouble. Back when I had a small webhost (I stopped when shared/reseller hosting descended too far into an overselling-and-deliberately-misleading-advertising-to-compete-on-price race to the bottom) a customer who "left" (was told to get lost due to non-payment) demanding I keep his content up because some users were still ending up there. As far as I could tell me of the records had ever had a TTL longer than the four hours )my default) while they were pointing at that address, yet more than a month later some traffic was still coming in to that address for that domain. I didn't look too deeply into it due to the history of the client in question, but it was certainly a real problem at that point in time.
So for my own stuff if there is a controlled change I try to keep the original address operating as a relay to the new address for a chunk of time, and for a while after that have it host a message saying "your DNS resolution seems to be broken, you shouldn't have been sent here, please report this to your ISP or local SysAdmin, if you want me to try fix it for you here is a list of my consulting fees".
(Weeping in agreement.)
The lesson I took away from hosting operations is that the implementation of internet standards has a long tail of customization; it's part of the job to accommodate them graciously. :)
Do you mean gracefully?
No reason one can't be both graceful and gracious.
Yes, thank you. (Past edit window.)
I've had a lot of issues with this for some of our customers. Nobody wants to run old deploy environments for weeks..
Turns out their routers somehow sets an insane TTL (like max int), and reports this to their clients, which in turn also get stuck with an insane TTL. You have to reboot or flush both the router and the clients to get it unstuck. I don't know if it's a "feature" or some sort of memory corruption / race condition.
The routers were almost always Asus routers, ex: Asus RT66. And the same customers were repeat offenders (not every time, but often enough).
In the end we had them set DNS on their computers to something like 1.1.1.1
I know that HTTP clients in some platforms like .NET will not resend the DNS query until the underlying TCP connection is closed. So if the server is using keep alive an you keep sending requests, you might actually end up using the same IP address for a long time.
I think this is actually the desirable behavior. If you've got very long-running connections and you want to force a switch-over you can always drop the connection even if you don't have some in-band mechanism. If you're constantly watching for a DNS change all it takes is one DNS failure to kill a connection (or all connections). In general transient network issues are probably going to be more common than IP changes on your infrastructure and issues caused by the former are harder to debug.
This is part of the cancer, though. People notice that even though their TTL is low, changes don't propagate, so they set it even lower in an attempt to compensate.
It goes both ways, a vicious circle.
TTLs aren't respected -> people try lowering them even more.
Lots of low TTLs -> people configure a minimum or ignore them.
As an infrastructure provider with limited connectivity, seeing lots of low DNS TTLs means I'll just configure a saner minimum (say 1h) and consider it not my problem.
As someone who overrides TTL's for all domains on my home network, I agree with this. I use Unbound DNS to query upstream servers over a VPN. I override min ttl to 20 minutes and that has never caused any issues as far as I can tell. I have been doing this for many years.
>ISP in Alaska told me that they would ignore all TTLs and set them all to 7 days
This is exactly why people set TTLs lower; they get ignored.
> One of my friends who ran an ISP in Alaska told me that they would ignore all TTLs and set them all to 7 days
If you’re hosting for customers the difference here is that this is your friends fault, not yours.
The question is how much percentage of your legitimate customers respect DNS TTLs. Amazon relies on DNS pretty much, this is how they shift traffic from one datacenter to another.
Even with short TTLs, I often see Facebook mobile app users lingering for days(!) on the old IP in the logs, long after all other traffic is gone.
I'm not really sure what's up with that, as no-one has ever reported the site not being reachable in the Facebook app after an IP change. Either it does some very aggressive caching, or something is pretending to be the Facebook app.