Comment by hn_go_brrrrr
5 days ago
It's not controversial, it's just not how the ecosystem works. There has never been an expectation that someone make a notification about impending crawling.
It might be nice if there were categories that well-behaved bots could follow, as noted above, but even then the problem exists for bots doing new things that don't fall into existing categories.
My complaint here isn't what they did. It's that they explain it as "here's how to opt out" when the information was too late to allow people to opt out.
I think that's disingenuous of them.
It's been common knowledge for anyone running a web server since 1994.
I don't think you are reading my posts in full.