Comment by hollerith

The people who've been in charge of the web (i.e., mostly the browser makers, but also the owners of the most popular sites) have made decisions that are IMHO severely anti-user. Although these anti-user design decisions have been accumulating for 30 years, users have had no alternative because all the content was on the web with way to get it other than to visit web sites with a web browser.

Now that there is an alternative (namely AI) people (including me) are flocking to the alternative. You want frame this as unethical bots versus ethically-acceptable human site visitors, but the main motivation for the use of scraping bots these days is to provide services (i.e, AI-based question answering) that users (like me) consider far superior to going directly to web sites for information because visiting web sites with a web browser is a frustrating tedious experience.