← Back to context

Comment by mkj

1 day ago

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.

Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!

But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.

So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.

9 comments

mkj

Reply
Barbing  1 day ago

30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?

  • mkj  1 day ago

    There are a lot of urls for the bots to follow. Yesterday's logs had 2.2M unique URLs out of 2.7M requests (more unique than I was expecting!)

    Typical traffic:

      https://hg.ucc.asn.au/dropbear-fuzzcorpus/annotate/default/fuzzer-preauth/23f6b74342011f35e657f06e9398ad339360d7e9 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/comparison/e67e5df225e5/fuzzer-verify/b5f57a2bb85ab7ef957e4979fd202b091de27bd8?revcount=8 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/72337e3dcccf/fuzzer-preauth_nomaths/6b1f390d26b08759983937727e10ab90aa7b03f2?revcount=15 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/4cdb539e4b20/fuzzer-preauth_nomaths/713e5cd4f73276f00a5d6685cc16d4565ad3815a?revcount=96 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/tip/fuzzer-client_nomaths/540778ea81a915af6e898fd9658c5d9d6594c40a?revcount=128 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/0ef1e51e583f/fuzzer-client/65ac44bdf4868866def70f40542dd1b5ca447310?revcount=30

    (I've moved and kept the fuzz corpus now, but other viewvc repos were suffering similarly)

  • psd1  17 hours ago

    I have compiled a list of vibecoders who have impressed me with their consideration for other people. Here it is:

  • inigyou  16 hours ago

    None of the above. They're just really retarded bots. Nobody knows why they exist or what they gain from really stupid scraping.