Comment by franze
9 hours ago
llms.txt is supported by 0 of the relevant ai providers and must be seen as harmful
.. as the webmaster implemented something that they might thought has an impact (false sense of impact), but has zero
so net gain negative
i consider such lists harmful - a good website is one that supports the goal of the website providers and its desired users (some of these users might be bots)
a bad website is a website that does everything for everyone just because
"The Unreasonable Effectiveness of Checklists" (https://rs.io/unreasonable-effectiveness-of-checklists/) comes to mind.
When I was younger I would have though the same. Now that I have more humility and less working memory, I think differently.
but in a checklist you include what actually you need to check, not everything and especially not stuff that is harmful l and/or has negative gain
Think of those public, generic lists as checklist's checklists. You should look at those items asking "will adding this to my checklist help achieving my goals?", and answer with a heavy bias towards "no".
You won't find generic lists that are well suited to your case, and you certainly won't find any flawless one. If you don't know the details about one of those items, you either go with "no" or learn them. But there is a lot of value on getting a list you can look at and discover something that you forgot.
>llms.txt is supported by 0 of the relevant ai providers
True, but it serves a other purpose, especially when the website is offering developer-oriented services. It's a single link you can give your AI agent and ask to "read this, understand it does, implement it".
Sure, you could just point it at docs.<service>.com but there might be bot protection, authentication, JS-heavy content etc.
So i feel llms.txt still has a purpose.