Comment by marginalia_nu

I can't imagine using anything other than a stream interface when dealing with web requests in a crawler.

You need that to protect against not only these types of shenanigans, but also large or slow responses.