Comment by TimorousBestie

21 hours ago

LLMs broadly speaking disregard robots.txt, so I don’t see why they would bother regarding this.

0 comments