Comment by microtonal
2 days ago
How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill).
Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.
> they will just move the work to the residential proxies that they have access to
The proxies I am familiar with do not offer arbitrary code execution. I think you're thinking of a botnet.
Regarding native code, the current crop of solutions seem to work well enough for now. Ultimately a challenge response protocol should be standardized and browsers should ship a native implementation. In the meantime WASM likely gets you close enough to native.
Regarding native code, the current crop of solutions seem to work well enough for now.
I just quoted a toot in another submission, adding it here since it is relevant:
We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.
https://social.anoxinon.de/@Codeberg/115033790447125787