Comment by HermanMartinus
3 days ago
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path).
All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
Amazing, I didn't know.
So it get even stranger, I am the only one reading those /llms.txt ...
I'm seeing quite a bit of request for these on my work's GitBook documentation site.
But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.
How is a static blog being scraped a problem? Do you not use a CDN?
> a blogging platform with around 80k blogs
But nah, I'm sure OP doesn't know about CDNs.
Are all blogs static though?
Very few blogs require frequent updates. Even with user comments.
> I can definitively say llms.txt is not used by any AI players.
OP clearly meant that the AI players are not reading and/or honouring llms.txt of other websites when scraping.
i stand corrected, but what was clear to you, obviously was not clear to me.