Comment by lavela
21 hours ago
"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more."
https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
You could try different compression methods supported by browsers like brotli.
Otherwise you can also chain compression methods like: "Content-Encoding: gzip gzip".
Ah cool that site's robots.txt is still broken, just like it was when it first came up on HN...