Comment by miki123211
1 month ago
As the post says, Google only scrapes the websites that want to be scraped. Sure, it's opt-out (via robots.txt) rather than opt-in, but they do give you a choice. You can even decide between no scraping at all and opting out on a per-scraper basis, and Google will absolutely honor your preferences in that regard.
SERP API just assumes everybody wants to be scraped, and doesn't give you a choice.
(whether websites should have such a choice is a different matter entirely).
This is a bad argument because Google is using its monopoly to effectively force websites to allow Google to scrape them.
requiring me to explicitly opt-out of something is NOT the same thing as getting my consent. So your argument breaks down there.
You know what getting my consent would look like? Google hosting a form where i can tell them PLEASE SCRAPE MY WEBSITE and include it in your search results. That is what consent looks like.
Google has never asked for my consent. Yet they expect others to behave by different rules.
Now where google may have a reasonable case is that google scrapes with the intention of offering the data “for free”. SerpAPI does not.
> You know what getting my consent would look like? Google hosting a form where i can tell them PLEASE SCRAPE MY WEBSITE and include it in your search results. That is what consent looks like.
Just for anybody wondering, they have always had such a form as well. Apart from their general crawling.
It's never been the case that if you put something into public, then you get to reserve your right to refuse public access. Either it's public and strangers can look at it. Or it's private and you need to implement a gate.
If this is about protecting third parties from being scraped, why does Google have an interest at all? Surely Google won't have the relevant third-party data itself because, as you say, Google respects robots.txt. So how can that data be scraped from Google?
I don't think this suit is actually about that, though. I think Google's complaint is that
> SerpApi deceptively takes content that Google licenses from others
In other words, this is just a good old-fashioned licence violation.