Comment by zzo38computer

7 months ago

I also had the idea of zip bomb to confuse badly behaved scrapers (and I have mentioned it before to some other people, although I did not implemented it). However, maybe instead of 0x00, you might use a different byte value.

I had other ideas too, but I don't know how well some of them will work (they might depend on what bots they are).

3 comments

zzo38computer

Reply
ycombinatrix  7 months ago

The different byte values likely won't compress as well as all 0s unless they are a repeating pattern of blocks.

An alternative might be to use Brotli which has a static dictionary. Maybe that can be used to achieve a high compression ratio.

  • zzo38computer  7 months ago

    I meant that all of the byte values would be the same (so they would still be repeating), but a different value than zero. However, Brotli could be another idea if the client supports it.

  • dspillett  7 months ago

    Compressing a sequence of any single character should give almost identical results length-wise (perhaps not exactly identical, but the difference will be vanishingly small).

    For example, with gzip using default options:

        me@here:~$ pv /dev/zero -s 10M -S | gzip -c | wc -c                    
    10.0MiB 0:00:00 [ 122MiB/s] [=============================>] 100%      
    10208                                                                  
    me@here:~$ pv /dev/zero -s 100M -S | gzip -c | wc -c                   
     100MiB 0:00:00 [ 134MiB/s] [=============================>] 100%      
    101791                                                                 
    me@here:~$ pv /dev/zero -s 1G -S | gzip -c | wc -c                     
    1.00GiB 0:00:07 [ 135MiB/s] [=============================>] 100%      
    1042069                                                                
    me@here:~$ pv /dev/zero -s 10M -S | tr "\000" "\141" | gzip -c | wc -c 
    10.0MiB 0:00:00 [ 109MiB/s] [=============================>] 100%      
    10209                                                                  
    me@here:~$ pv /dev/zero -s 100M -S | tr "\000" "\141" | gzip -c | wc -c
     100MiB 0:00:00 [ 118MiB/s] [=============================>] 100%      
    101792                                                                 
    me@here:~$ pv /dev/zero -s 1G -S | tr "\000" "\141" | gzip -c | wc -c  
    1.00GiB 0:00:07 [ 129MiB/s] [=============================>] 100%      
    1042071

    Two bytes difference for a 1GiB sequence of “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa…” (\141) compared to a sequence of \000.