Comment by justinsaccount
7 years ago
There was a dns record looked up primarily by large supercomputers that had a 0 ttl. It was used for stats via a UDP packet (because it was non blocking, nevermind that the dns query was blocking). This was set to 0 for "failover" but it hadn't changed in years. I worked out that our systems alone had caused billions of queries for this name.
After I complained I think they upped the ttl.. to 60.
Reminds me a of a server pair at the last healthcare place I worked. Between the two of them they'd generate something around 1200 DNS lookups per second (about 60% of the load on the DNS servers) of their own name. I think the logic was if the name stopped responding then server A was primary. If the name was responding the server that owned the IP it was responding for was primary. If the servers wanted to swap primary/secondary they would issue a DDNS request.
After about 8 years we were were restructuring our DNS infrastructure for performance and I rate limited those two to 10 or so queries per second each. In that time there must have been 300 billion or so requests from those two boxes alone.
In my experience that sort of thing is from the local hostname not being present in /etc/hosts and (of course) a caching resolver not in use.
Some process on the system wants to connect to itself, which then causes a dns lookup. Add a high transaction rate on top of that, and 1,200/second is easy.
The funniest one I remember seeing was thousands and thousands of lookups for HTACCESS. Turned out apache was running on top of a web root stored in AFS and not configured to stop looking for .htaccess files at the project root, so it would try to open
When it hit /afs/.htaccess then it would try to contact the AFS sell called .htaccess, which would do a bunch of dns lookups.
This would be done something like twice for every incoming web request.