Slacker News Slacker News logo featuring a lazy sloth with a folded newspaper hat
  • top
  • new
  • show
  • ask
  • jobs
Library
← Back to context

Comment by alt187

5 months ago

The tarpit is made for LLM crawlers who don't respect robots.txt. Do you love LLMs so much that you wish that they wouldn't have to respect this stupid, anticorporate AI-doomer robots.txt convention so they can pry out of the greedy hands of the webserver one more URL?

Maybe you just had a knee-jerk reaction.

0 comments

alt187

Reply

No comments yet

Contribute on Hacker News ↗

Slacker News

Product

  • API Reference
  • Hacker News RSS
  • Source on GitHub

Community

  • Support Ukraine
  • Equal Justice Initiative
  • GiveWell Charities