Comment by TonyTrapp
10 hours ago
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
Which, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
Makes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.