Comment by threecheese
10 hours ago
> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request.
That’ll do it.
Ahh, the three relevant numbers in development: 0, 1, and infinity.
The incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.
This is a pretty interesting solution. I could see how this could useful for certain kinds of problems (as part of a ddos attack mitigation for example).
An oldie and a goodie
https://news.ycombinator.com/item?id=21865715
I mean, it's one GetPostRecord, Michael. What could it cost? 1 trillion ports?
Zero, one, many, many thousands.
And then they fix the issue by using multiple localhost IPs rather than, perhaps, not sending 15-20 thousand URIs at a time
They mentioned it was a temporary fix that they removed after finding and fixing the true root cause, though.
less than ideal if I had to be frank.