Comment by Nextgrid
2 months ago
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.
SerpApi doesn't have that privilege.
2 months ago
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.
SerpApi doesn't have that privilege.
Some domains do block Google, often partially. There are some statistics here:
https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...
Google has respected robots.txt from the start.
but SerpApi is not scraping websites, it is sending malicoius requests to google.com.
SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.