Comment by Someone

Do scrapers actually do such things on every page they download? Sampling a small fraction of a site to check how trustworthy it is, I can see happen, but I would think they’d rather scrape many more pages than spend resources doing such checks on every page.

Or is the internet so full of garbage nowadays that it is necessary to do that on every page?