Comment by ricardo81
2 months ago
Unfortunately they do have a couple of points that may prove salient (though I fully agree about them being scrapers also).
You can search Google _for free_ (with all the caveats of that statement), part of their grievance is that serpapi use the scraped data as a paid for service
Lots of Google bot blocking is also circumvented, which they seem to have made a lot of efforts towards in the past year
- robots.txt directives (fwiw)
- You need JS
- If you have no cookie you'll be given a set of JS fingerprints, apparently one set for mobile and one for desktop. You may have to tweak what fingerprints you give back in order to get results custom to user agent etc.
Google was never that bothered about scraping if it was done at a reasonable volume. With pools of millions of IPs and a handle on how to get around their blocking they're at the mercy of how polite the scraping is. They're maybe also worried about people reselling data en masse to competitors i.e. their usual all your data belongs to us and only us.
> You can search Google for free
I thought the ads counted as payment? That seems to be the logic used to take technical measures against adblockers on YouTube while pushing users towards a paid ad-free subscription, at least.
If viewing ads is payment, then Google isn't a free service. If viewing ads isn't payment, then Google should have no problem with people using adblockers.
I don't disagree with the logic and it definitely is/was their business model, scraping/crawling the web and subsidising the service with ads. But clicking on ads are optional.
No google's business model is showing you ads, not clicking on them. That's the job of the person who designs the ad.
Google would like you to click through as it looks better for their stats, but they don't actually care.
> You can search Google _for free_
Well not through their API which you do need to pay for and is a paid service.