Comment by mrweasel

7 months ago

First of: Google has not once crashed one of our sites with GoogleBot. They have never tried to by-pass our caching and they are open and honest about their IP ranges, allowing us to rate-limit if needed.

The residential proxies are not needed, if you behave. My take is that you want to scrape stuff that site owners do not want to give you and you don't want to be told no or perhaps pay a license. That is the only case where I can see you needing a residential proxies.

9 comments

mrweasel

Reply
TZubiri  7 months ago

>The residential proxies are not needed, if you behave

I'm starting to think that somee users in hackernews do not 'behave' or at least they think they do not 'behave' and provide an alibi for those that do not 'behave'.

That the hacker in hackernews does not attract just hackers as in 'hacking together features' but also hackers as in 'illegitimately gaining access to servers/data'

As far as I can tell, as a hacker that hacks features together, resi proxies are something the enemy uses. Whenever I boot up a server and get 1000 log in requests per second and requests for commonly exploited files from russian and chinese IPs, those come from resi IPs no doubt. There's 2 sides to this match, no more.

tonymet  7 months ago

You can’t get much crawling done from published cloud IPs. Residential proxies are the only way to do most crawls today.

That said, I support Google working to shut these networks down, since they are almost universally bad.

It’s just a shame that there’s no where to go for legitimate crawling activities.

  • mrweasel  7 months ago

    > You can’t get much crawling done from published cloud IPs.

    Think about why that might be. I'm sorry, if you legitimately need to crawl the net, and do so from a cloud provide, your industry screwed you over with bad behaviour. Go get hosting with a company that cares about who their customers are, you're hanging out with a bad crowd.