Comment by dathinab

it's indeed not a "bot/crawler protection"

it's a "I don't want my server to be _overrun_ by crawlers" protection which works by

- taking advantage that many crawlers are made very badly/cheaply

- increasing the cost of crawling

thats it, simple but good enough to shake of the dumbest crawlers and to make it worth it for AI agents to e.g. cache site crawling so that they don't craws your site a 1000 times a day but instead just once