Comment by gwern

You don't even need to do git or a tarball! HTTP/HTML already has an "API" for serving Markdown to any agent like a LLM which wants it, because you can easily set a server to return Markdown with an accept-encoding (kinda why that functionality exists in the first place).

I set my nginx to return the Markdown source (which is just $URL.md) for my website; any LLM which wants up-to-date docs from my website can do so as easily as `curl --header 'Accept: text/markdown' 'https://gwern.net/archiving'`. One simple flag. Boom, done.