Comment by ksec
21 hours ago
Is there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS?
Or they could at least have the curtesy to scrap during night time / off peak hours.
21 hours ago
Is there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS?
Or they could at least have the curtesy to scrap during night time / off peak hours.
No, because they won't pay for anything they can get for free. There's only one situation where an AI company will pay for data, and that's when it's owned by someone with scary enough lawyers to pressure them into paying up. Hence why OpenAI has struck licensing deals with a handful of companies while continuing to bulk-scrape unlicensed data from everyone else.
There is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/
Is existing intellectual property law not sufficient? Why aren't companies being prosecuted for large-scale theft?