Comment by johnnyanmac

>Why this is the case while web-crawlers have been scrapping the web for the last 30 years is a mystery to me.

a mix of ignorance, greed, and a bit of the tragedy of the commons. If you don't respect anyone around you, you're not going to care about any rules or ettiquite that don't directly punish you. Society has definitely broken down over the decades.