Comment by ThinkBeat

How many TB is your repo?

Do they keep retrieving the same data from the same links over and over and over again, like stuck in a forever loop, that runs week after week?

Or are they crawling your site at a hype aggressive way but getting more and more data? So it may tea them last say 2 days to crawl over it and then they go away?