Comment by esseph
4 hours ago
Data scraping.
If anything else, the AI machine wants near constant streams of new data, even if it already checked with you 30ms ago.
A CDN helps immensely.
Also keeps you from getting DDoS'd if you did something like run it off your home connection.
Your webserver most likely have "rate limiting" built in already, which you can configure to act based on lots of variables typically. Set a limit of 1 req/s or whatever, and you've stopped 99% of all DDoS you'll encounter on the public web. If your visitors get cranky, up it to 10 req/s and you still are preventing most of the "abusive traffic", granted your backend/website isn't completely upside down when it comes to performance and resource usage.
CDN is something you do once you run out of options, not something you should reach for immediately, it makes no sense in most cases of just hosting a website.
This is not how things work and no company providing online services for money would rate limit like that. This would do nothing for real world DDoS. You're in "not even wrong" territory.
And for large services implementing a CDN properly takes days/weeks of preparation. Once you're down it's way too late.
Yeah, of course if you're a large service, stuff that works for SMEs isn't gonna work for you...
If your problem is AI crawlers, then simple rate limits help, I've helped countless of businesses with this already. For the ones that it isn't enough, you continue adding more roadblocks. There is no "one size fits all here" and that you seemingly is under that belief, leads less credence to what you're saying, not more.