Comment by jcynix

17 hours ago

If you control your own Apache server and just want to shortcut to "go away" instead of feeding scrapers, the RewriteEngine is your friend, for example:

      RewriteEngine On

      # Block requests that reference .php anywhere (path, query, or encoded)
      RewriteCond %{REQUEST_URI} (\.php|%2ephp|%2e%70%68%70) [NC,OR]
      RewriteCond %{QUERY_STRING} \.php [NC,OR]
      RewriteCond %{THE_REQUEST} \.php [NC]
      RewriteRule .* - [F,L]

Notes: there's no PHP on my servers, so if someone asks for it, they are one of the "bad boys" IMHO. Your mileage may differ.

7 comments

jcynix

Reply
palsecam  13 hours ago

I do something quite similar with nginx:

  # Nothing to hack around here, I’m just a teapot:
  location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { 
      return 418; 
  }
  error_page 418 /418.html;

No hard block, instead reply to bots the funny HTTP 418 code (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...). That makes filtering logs easier.

Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is WordPress login URL, and it’s commonly blindly requested by bots searching for weak WordPress installs.)

  • kijin  12 hours ago

    nginx also has "return 444", a special code that makes it drop the connection altogether. This is quite useful if you don't even want to waste any bandwidth serving an error page. You have an image on your error page, which some crappy bots will download over and over again.

    • quesera  2 hours ago

      Beware of nginx 444 if your webserver is behind a load balancer.

      The LB will see the unresponded requests and think your webserver is failing.

      Ideal would be to respond at the webserver and let the LB drop the response.

    • palsecam  11 hours ago

      Yes @ 444 (https://http.cat/status/444). That’s indeed the lightest-weight option.

      > You have an image on your error page, which some crappy bots will download over and over again.

      Most bots won’t download subresources (almost none of them do, actually). The HTML page itself is lean (475 bytes); the image is an Easter egg for humans ;-) Moreover, I use a caching CDN (Cloudflare).

    • MadnessASAP  5 hours ago

      Does it also tell the kernel to drop the socket? Or is a TCP FIN packet still sent?

      Be better if the scraper is left waiting for a packet that'll never arrive (till it times out obviously)