Comment by bluGill
2 months ago
At this point everyone knows about robots.txt, so if you didn't opt-out that is your own fault. Opting out of everyone at once is easy, and you get fine grained control if you want it.
Also most people would agree they are fine with being indexed in general. That is different from email spam where people don't want it.
Looking at SerpApi clients, looks like most companies would agree they are fine with scraping Google. That is different from having your website content stolen and summarized by AI on Google search, which people don't want.
The claim is SerApi is not honoring robots.txt, and they are getting far more data from google/more often than needed for an index operation. Or at least that is the best I can make out of the claim in court from the article - I have not read the actual complaint.
People are generally fine with indexing operations so long as you don't use too much bandwidth.
Using AI to summarize content is still and open question - I wouldn't be surprised if this develops to some form of "you can index but not summarize", but only time will tell.
Or by Google codewiki, which is morally the equivalent to making a business out of ersatz travel guides by ripping off the authors of real ones