Can you please stop fulminating and posting flamebait and/or unsubstantive comments to HN threads? All of that is against the guidelines and you have unfortunately been doing them repeatedly.
You realize humans are going to be the first wave of collateral damage right? I already basically cannot browse the internet for technical information, since most high-quality forums are behind captchas that block my iPhone.
If I ask an agent to do it, it does better at finding the small percentage of sources not hosted by cloudflare. However, it generally cannot hit open-access / public domain sources (like the current legal code, or academic papers) because those are blocked and it respects stuff like robots.txt.
Can't speak for GP, but I wouldn't - privacy is already eroding at a startling rate, and more KYC for things that really don't need it is just a further affront to human rights. (See also the FCC's recent request for comments on requiring government-issued ID to use a cell phone.)
i installed the playwright MCP to let my agent access walled sites (specifically ebay and WSJ). i noticed that 90% of the time it was bounced from a site, it just reached out to a different site that wasn't walled, and i think it's the right move: most information exists at multiple places on the web, it's cheaper and _faster_ to just skip over walled sources.
for the forum example: many forums have a policy to only allow access to attachments to logged-in users. i can't remember the last time i registered at a new forum just to view an attachment: the effect has always been to drive me elsewhere. no complaints -- these solutions work if your goal is to reduce load. i'm suspicious that they can drive monetization outside of a very few niches.
I thought the goal was to only charge agents a fee, which would either 1. stop agents from scraping your site non-stop and eliminate the need for a captcha, making the human experience better or 2. make the owner of the site some money in exchange for a bajillion bots scraping their content.
Maybe that's too optimistic though based on the responses in this thread.
I play dungeon crawl stone soup (think nethack,but with web tiles), and most of the servers are struggling because of AI crawlers downloading the morgues.
Real users are already suffering.
If (big if) the AI labs can be made to pay for the abuse, actual users win.
Can you please stop fulminating and posting flamebait and/or unsubstantive comments to HN threads? All of that is against the guidelines and you have unfortunately been doing them repeatedly.
If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
when the law won't protect you it creates an opportunity for a mafia like protection racket
You realize humans are going to be the first wave of collateral damage right? I already basically cannot browse the internet for technical information, since most high-quality forums are behind captchas that block my iPhone.
If I ask an agent to do it, it does better at finding the small percentage of sources not hosted by cloudflare. However, it generally cannot hit open-access / public domain sources (like the current legal code, or academic papers) because those are blocked and it respects stuff like robots.txt.
Would you be willing for Cloudflare to "Know their customer" (you) and pay 3 cents to access the forum, instead of filling in the captcha?
Can't speak for GP, but I wouldn't - privacy is already eroding at a startling rate, and more KYC for things that really don't need it is just a further affront to human rights. (See also the FCC's recent request for comments on requiring government-issued ID to use a cell phone.)
2 replies →
i installed the playwright MCP to let my agent access walled sites (specifically ebay and WSJ). i noticed that 90% of the time it was bounced from a site, it just reached out to a different site that wasn't walled, and i think it's the right move: most information exists at multiple places on the web, it's cheaper and _faster_ to just skip over walled sources.
for the forum example: many forums have a policy to only allow access to attachments to logged-in users. i can't remember the last time i registered at a new forum just to view an attachment: the effect has always been to drive me elsewhere. no complaints -- these solutions work if your goal is to reduce load. i'm suspicious that they can drive monetization outside of a very few niches.
I thought the goal was to only charge agents a fee, which would either 1. stop agents from scraping your site non-stop and eliminate the need for a captcha, making the human experience better or 2. make the owner of the site some money in exchange for a bajillion bots scraping their content.
Maybe that's too optimistic though based on the responses in this thread.
2 replies →
I play dungeon crawl stone soup (think nethack,but with web tiles), and most of the servers are struggling because of AI crawlers downloading the morgues.
Real users are already suffering.
If (big if) the AI labs can be made to pay for the abuse, actual users win.