← Back to context

Comment by semiquaver

8 days ago

But it doesn’t cost scrapers millions of solved challenges to go to millions of webpages on a single origin. Once you solve an Anubis challenge you get a signed JWT that lets you scrape a given site an unlimited amount for a configurable amount of time. (~day). So in practice it doesn’t actually cost the scrapers a large amount amount in proportion to their usage. It actually costs them proportionally less than a normal human.

To actually make it expensive for scrapers every page would need a new challenge. And that would not be tolerated by real human users. Or the challenge solution would need to be tied to a stateful reward that only entitles a human-level amount of subsequent request usage.

Well, getting that JWT gives you something to track requests by. So you can count how many requests you get per jwt,then if that token is making too many requests you can block it, throttle it,tarpit it,etc.

I'm not sure if anubis currently does that, but it certainly could.