Comment by dawnerd
2 days ago
Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.
Interesting. Now they seem to claim that not only they follow robots.txt for crawling, but that they also broke under pressure and made the unfortunate decisions to have user requests follow robots.txt too.
https://www.perplexity.ai/de/hub/technical-faq/how-does-perp...
User Agent block just means they'd spoof their user agent.