Comment by colinsane

> I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit.

Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:

``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```

another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.

``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```