Comment by scotty79
16 hours ago
If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up so it never exceeds what's comfortable for you. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000 a second currently at very little cost to them.
The module doing proof of work might be a part of your frontend that is static and on CDN.
If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.
I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.
The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.
Proof of work is bad rate limiting. The power asymmetry is too immense. Worst-case example is SHA-256, where an attacker with dedicated hardware is millions of times as efficient as common users. Even with a less boneheadedly foolish POW scheme (quite incidentally, every single web POW scheme that I know of started with SHA-256, and almost all still use it), the ratio is easily in the hundreds or thousands, too great an imbalance to bridge. (That is: if you want to require an attacker to burn one second of computation time, some cheap smartphone user will now be stuck for five minutes. If you reduce the difficulty so that user is only stopped for five seconds, your attacker is now burning less than 20ms.)
Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.