Comment by latenightcoding

some of these companies are straight up inept. Not an AI company but "babbar.tech" was DDOSing my site, I blocked them and they still re-visit thousands of pages every other day even if it just returns a 404 for them.