Ask HN: Non-Malicious Bots
11 days ago
When I ask an AI to load a link, sometimes it can load it, and sometimes it can't (and I have to copy paste the text myself).
Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop?
Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP requests?
Most of the times it's just a check that the bot is not making too many request, sometimes using a recent request list, sometimes asking the browser to make a very hard calculation that takes a few seconds to slow down everyone. It does not distinguish good bots and bad bots, just try to protect the server against too many requests.