Comment by Dylan16807
14 hours ago
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.
14 hours ago
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.
No comments yet
Contribute on Hacker News ↗