← Back to context

Comment by SR2Z

16 hours ago

IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.

4 comments

SR2Z

Reply
prinny_  15 hours ago

Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.

  • compootr  14 hours ago

    OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web

    there is a case to be made about the value of the traffic you'll get from oai search though...