Comment by bilekas
1 day ago
Because of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
1 day ago
Because of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
If bandwidth cost is a concern the maybe you should reconsider how you publish your site.
Like, what if you actually post something that gains traction, is it going to bankrupt you or something?
It's not just financial, you're taking up a lot of bandwidth, resources etc.
It's not just some light bump in traffic. It's a headache that shouldn't need to be dealt with if they would respect ROBOTS.txt. Quite simple really.