Comment by recursive
15 hours ago
If a new directive was introduced that allows for an explicit setting in robots.txt, do you think the bros would follow it anyway? Something like `ALLOW AGENTS` or `DISALLOW AGENTS`
15 hours ago
If a new directive was introduced that allows for an explicit setting in robots.txt, do you think the bros would follow it anyway? Something like `ALLOW AGENTS` or `DISALLOW AGENTS`
The "Bros"? Maybe.
I wouldn't want them to. The whole point of using agents to do stuff on the web for me, is for them to do the stuff on the web for me.
This is the reverse of "do not track" case. It'll not be effective because every service will set it to DISALLOW by default anyway, because it costs them nothing, and for most services, it actually is what they want anyway - most of businesses on the web are making money on wasting people's time, and for that, they need to force themselves on people; end-user automation defeats that, so they actively fight it (and complain a lot).