Comment by bradly
17 hours ago
Do you think there should be a way for a site to tell an agent it isn't allowed access? I'm not sure where I land on this exactly tbh, so no judgment cast.
17 hours ago
Do you think there should be a way for a site to tell an agent it isn't allowed access? I'm not sure where I land on this exactly tbh, so no judgment cast.
robots.txt?
robots.txt is a shitshow just like user agents. It's been twisted so many ways it doesn't reliably signal actual intent any more.
Has it been settled whether robots.txt applies to user-driven chat sessions and if things like the crawl delay should be applied to say an end-user, an ip address, a harness provider, etc? My understanding is robots.txt is more for training exclusions, but less so for agent work.
robots.txt was only intended to help search index crawlers not get stuck in endless crawl loops for badly designed websites.
What you suggest is explicitly not a purpose of robots.txt per RFC9309[1]:
"These rules are not a form of access authorization."
HTTP 429 and HTTP 403 are what servers are meant to return to clients to slow them down or tell them to stop doing something without having first gained authorisation.
[1] https://datatracker.ietf.org/doc/html/rfc9309#section-1
robots.txt applies (or should, in my opinion) to anything that automatically follows a link. Basically any software that is not a human-controlled web browser or single-shot curl command. Everything else: robot.
AI bros think they should be exempt from robots.txt. Administrators of big services beg to differ. No solid consensus has arisen. I bet it's gonna take a lawsuit or two to see how it shakes out.
4 replies →
I want agents to be able to act on my behalf, that’s the entire point. An agent should be able to do anything I can do sitting at my browser.
I suspect most people would be ok with this if they could only do it at the rate and frequency you yourself can do it. The problem is largely one of scale.
Is scale what we're discussing though?
e.g. a prompt of "fetch <article URL> and summarise it for me" is very close to what a human would be doing with a web browser, and doesn't seem to involve any kind of scaling issue.
3 replies →
Then make agent friendly content. Take the text and make a markdown version.
4 replies →
Technically, even your browser is an agent. It says it in the HTTP: User-Agent. So is cURL. Every application the user runs is acting on the user's behalf.
Dunno why the downvotes. I feel that is reasonable as well. Owners that block that stuff are doing so only to their detriment.