With a tuned cool down period this isn't a problem, especially if you frequent the sites. OpenWRT uses Anubis and usually when I need to peruse their site I'm on a very low-end device. I prefer waiting much more over finding Waldos
But in principle I agree that there's no good answer to this, scraping _is_ useful and I bet most of us here had scraped something, it is AI company and their use of human's material for training without consent and return that led us to this (I know botting exists in forum since forum is a thing but it is easily solved by human moderators and keyword filter)
But after you’ve completed the Anubis PoW challenge for a site, it remains valid for some amount of time.
So it’s not quite as horrible as it sounds.
I have setting up Anubis for my own sites on my todo list. And I wish more people did it too. I don’t really mind waiting a little bit extra every now and then before the page loads. What I do mind is ReCaptcha asking me to click all the pictures with buses in them etc. And especially when I have to do it several times over before it’s happy. I’d rather wait a minute for a page to load than to ever solve a ReCaptcha again, if given the choice.
One of unexpected outcomes from AI-induced hardware shortage may be that, in fact, compute won’t be getting cheaper and may in fact get more expensive…
Bots don't [currently] execute JavaScript or follow complicated redirects.
They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.
It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.
With a tuned cool down period this isn't a problem, especially if you frequent the sites. OpenWRT uses Anubis and usually when I need to peruse their site I'm on a very low-end device. I prefer waiting much more over finding Waldos
But in principle I agree that there's no good answer to this, scraping _is_ useful and I bet most of us here had scraped something, it is AI company and their use of human's material for training without consent and return that led us to this (I know botting exists in forum since forum is a thing but it is easily solved by human moderators and keyword filter)
Anubis often takes more than 60 seconds to complete on low-end devices (especially old smartphones). It seems like there's no good solution.
But after you’ve completed the Anubis PoW challenge for a site, it remains valid for some amount of time.
So it’s not quite as horrible as it sounds.
I have setting up Anubis for my own sites on my todo list. And I wish more people did it too. I don’t really mind waiting a little bit extra every now and then before the page loads. What I do mind is ReCaptcha asking me to click all the pictures with buses in them etc. And especially when I have to do it several times over before it’s happy. I’d rather wait a minute for a page to load than to ever solve a ReCaptcha again, if given the choice.
That must be really low end then. I’ve never seen it complete in a timeframe that was slower than “I can’t even read the page before it redirects”
There's not an easy, perfect solution, for sure. Newer phones get faster, but spammer compute gets cheaper.
Some sort of decentralized trust web seems like another option, though less viable.
One of unexpected outcomes from AI-induced hardware shortage may be that, in fact, compute won’t be getting cheaper and may in fact get more expensive…
How does Anubis stop bots?
Bots don't execute JavaScript or follow complicated redirects.
Bots don't [currently] execute JavaScript or follow complicated redirects.
They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.
It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.