Comment by NobodyNada
15 hours ago
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
15 hours ago
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.
Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.
It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
> In other words not a meaningful barrier.
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
2 replies →
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently.
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
1 reply →
...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of?