Comment by hiccuphippo
8 days ago
I wonder if the crawlers are pretending to be something else to avoid getting blocked.
I see Bun (which was bought by Anthropic) has all its documentation in llms.txt[0]. They should know if Claude uses it or wouldn't waste the effort in building this.
As a project that started with a lot of idealism about how software _should_ be built, I would totally expect Bun to have an llms.txt file even if Claude wasn't using it. It's a project that is motivated in part by leading by example.
I also noticed this LLMs.txt at bun.sh, so for me it looks like some sort of advertising.
Optimistic to assume the Bun team and the Claude team talk to each other
Did they do that before they were bought by Anthropic? Perhaps it's just part of a CI process that nobody's going to take an axe to without good reason.