Comment by AnthonyMouse

> So there is no benefit to them behaving.

That's assuming they're deriving a benefit from misbehaving.

There is no benefit to immediately re-crawling 404s or following dynamic links into a rabbit hole of machine-generated junk data and empty search results pages in violation of robots.txt. They're wasting the site's bandwidth and their own in order to get trash they don't even want.

Meanwhile there is an obvious benefit to behaving: You don't, all by yourself, cause public sites to block everyone including you.

The problem here isn't malice, it's incompetence.