Comment by mvdtnz

I mean if they want to "wriggle out" of robots.txt they can just ignore it. It's entirely voluntary.