Comment by idontsee
2 days ago
> There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant.
It isn't just the volume of requests, but also bandwidth. There have been cases where scraping represents >80% of a forge's bandwidth usage. I wouldn't want that to happen to the one I host at home.
Sure but how much bandwidth is that actually? Of course if your normal traffic is pretty low, it's easy for bot traffic to multiply that by 5, but it doesn't mean it's actually a problem.
The market price for bandwidth in a central location (USA or Europe) is around $1-2 per TB and less if you buy in bulk. I think it's somewhat cheaper in Europe than in the USA due to vastly stronger competition. Hetzner includes 20TB outgoing with every Europe VPS plan, and 1€/TB +VAT overage. Most providers aren't quite so generous but still not that bad. How much are you actually spending?