Comment by JohnFen

Sadly true. That's why I gave up on robots.txt years ago and started blocking crawlers outright in .htaccess

Of course, that became unsustainable so now I have everything behind a login wall.