Comment by jeremyjh

4 hours ago

The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.

But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.