We are developing a web application security scanner [1] and we indeed use max length setting and also detect binary responses, just tested this and as expected it worked fine.
I'm actually surprised that many other scanners failed to do this.
> Wouldn't all but the most naive scanners use time-out settings, maximum lengths on bytes read etc?
It wouldn't save a scanner from crashing to use a time-out or max read bytes. The defense can send the 100kb zipped data in a matter of seconds. The client then decompresses the zipped data which expands to gigabytes, causing crashes by out-of-memory.
User ruytlm has posted links to hacker factor blog, and it seems some sophisticated scanners (e.g., Eddie) were crashed by the exploit. In that blog the author postulates that Eddie is a nation-state level (not script kiddie) scanner, so I'd say that the answer to this question will be in your definition of naive. It's tempting to qualify any scanner which crashes on this as naive though, I'd agree. Especially moving forward with the publicity of this post/topic.
Well actually from memory the author of the blog was doubtful if this exploit actually crashed Eddie or not, but it did crash the other bots (Eddie V1 did go offline, possibly as a crash), so it would appear you are correct. Only truely naive bots might well be affected by this.
We are developing a web application security scanner [1] and we indeed use max length setting and also detect binary responses, just tested this and as expected it worked fine.
I'm actually surprised that many other scanners failed to do this.
[1] https://www.netsparker.com
> Wouldn't all but the most naive scanners use time-out settings, maximum lengths on bytes read etc?
It wouldn't save a scanner from crashing to use a time-out or max read bytes. The defense can send the 100kb zipped data in a matter of seconds. The client then decompresses the zipped data which expands to gigabytes, causing crashes by out-of-memory.
Was thinking more about a maximum length for the decompression stage.
User ruytlm has posted links to hacker factor blog, and it seems some sophisticated scanners (e.g., Eddie) were crashed by the exploit. In that blog the author postulates that Eddie is a nation-state level (not script kiddie) scanner, so I'd say that the answer to this question will be in your definition of naive. It's tempting to qualify any scanner which crashes on this as naive though, I'd agree. Especially moving forward with the publicity of this post/topic.
Well actually from memory the author of the blog was doubtful if this exploit actually crashed Eddie or not, but it did crash the other bots (Eddie V1 did go offline, possibly as a crash), so it would appear you are correct. Only truely naive bots might well be affected by this.