Comment by kassner

I can't comment on LLMs, but I vaguely remember speculations of it being a mixture of AI training and trying to keep resource utilization at bay. Even generic bots can cause a lot load if you don't have much caching in front of it.