Comment by cpncrunch

Actually, it looks like all the major ones do honour robots.txt including perplexity. They seemingly get around it using google serps, so theyre not actually crawling or hammering the site servers (or even cloudflare).

https://www.ailawandpolicy.com/2025/10/anti-circumvention-re...