Comment by TeMPOraL

> A lot of the internet is built on trust.

Yes. In this context, the problem is that you cannot trust websites to provide a standardized bulk download options. Most of them have (often pretty selfish or user-abusive) reasons not to provide any bulk download, much less proactively conform to some bottom-up standards. As a result, unless one is only targeting one or few very specific sites, even thinking about making the scrapper support anything but the standard crawling approach costs more in developer time than the benefit it brings.