Comment by miraline
14 hours ago
The interesting part is that bots mitigation is actually an economic problem. You don't need to make scraping impossible, you only need to just make it expensive enough so that abusive traffic stops being worth it.
14 hours ago
The interesting part is that bots mitigation is actually an economic problem. You don't need to make scraping impossible, you only need to just make it expensive enough so that abusive traffic stops being worth it.
Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.
Scrapers scrape every link they can find.
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
The main reason to have those links is for people to click on them. No sensibly programmed machine is scraping diffs from a git web front end.
2 replies →
> why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
I document my code, in issue and elsewhere on the Web, by referring to specific commits so having a URL for each is useful to me at least.