Comment by grishka
3 months ago
What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.
3 months ago
What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.
"What about AI not respecting robots.txt?"
since when actor that want gather your entire data respect things like this??? how can you enforce such things with just "please don't crawl this directory thanks"
> how can you enforce such things
A van, some balaclavas and 4 people with big sticks.
You can't enforce them, but for the entire preceding history of the internet, most crawlers respected them. But then AI happened and those companies decided that their noble mission of forcing their slop into as many facets of human life as possible is above some stupid rules.