Comment by MATTEHWHOU

The interesting thing about llms.txt isn't the file format — it's the incentive shift.

With robots.txt, you were telling crawlers to go away. With llms.txt, you're inviting them in and curating what they see. That's a fundamentally different relationship.

I've been experimenting with this on a few projects and the biggest lesson: your llms.txt should NOT be a sitemap. It should be the answer to "if an AI could only read 5 pages on my site, which 5 would make it actually useful to end users?"

The projects where I got this right saw noticeably better AI-generated answers about our tools. The ones where I just dumped every doc link? No difference from not having it at all.