Comment by A1kmm

If you use nginx to front it, consider something like this in the `http` block of your config:

    map $http_user_agent $bottype {
        default          "";
        "~.*Amazonbot.*" "amazon";
        "~.*ImagesiftBot.*" "imagesift";
        "~.*Googlebot.*" "google";
        "~.*ClaudeBot.*" "claude";
        "~.*gptbot.*" "gpt";
        "~.*semrush.*" "semrush";
        "~.*mj12.*" "mj12";
        "~.*Bytespider.*" "bytedance";
        "~.*facebook.*" "facebook";
    }
    limit_req_zone $bottype zone=bots:10m rate=6r/m;
    limit_req zone=bots burst=10 nodelay;
    limit_req_status 429;

You can still have other limits by IPs. 429s tends to slow the scrapers, and it means you are spending a lot less on bandwidth and compute when they get too aggressive. Monitor and adjust the regex list over time as needed.

Note that if SEO is a goal, this does make you vulnerable to blackhat SEO by someone faking a UA of a search engine you care about and eating their 6 req/minute quota with fake bots. You could treat Google differently.

This approach won't solve for the case where the UA is dishonest and pretends to be a browser - that's an especially hard problem if they have a large pool of residential IPs and emulate / are headless browsers, but that's a whole different problem that needs different solutions.