Comment by finnlab
3 days ago
Nice work, but I feel like it's not required to use AWS for this. There are small hosting companies with specialized servers (50gbit shared medium for under 10$), you could probably do this under 100$ with some optimization.
I did some crawling on hetzner back in the day. They monitor traffic and make sure you don't automate publically available data retrieval. They send you an email telling you that they are concerned because you got the ip blacklisted. Funny thing is: They own the blacklist that they refer to.
This. I tried to run a very slow DHT scraper I was writing on a Hetzner server and within minutes they were on my ass. I don't want to make an enemy of them so I killed it immediately, but they are clearly very sensitive to anything outside of "normal".
If Hetzner actually puts their own customers on their blacklist then that list becomes more trustworthy.
They were right to blacklist you, they were right to complain to you, and they were right not to assume malice and kick you off their platform/shut down your server.
Yes I wasn't banned or anything, they aren't barbarians. Also, explain your opinion don't just put it out there. This is not a football match.
2 replies →
This. AWS is like a cash furnace, only really usable for VC backed efforts with more money than sense.