Comment by ryanjshaw
3 hours ago
> This does nothing against your ability to scrape the web the Google way
I thought that Google has access to significant portions of the internet that non-Google bots won’t have access to?
3 hours ago
> This does nothing against your ability to scrape the web the Google way
I thought that Google has access to significant portions of the internet that non-Google bots won’t have access to?
Their crawler has known IPs that get a white-glove treatment by every site with a paywall for example