Comment by pjullrich
1 day ago
Yes, we can't block models from scraping the public repos and train on them, but *we* won't use your code (neither public nor private) to train models on them. We really only want to host your code and do that well.
1 day ago
Yes, we can't block models from scraping the public repos and train on them, but *we* won't use your code (neither public nor private) to train models on them. We really only want to host your code and do that well.
> we can't block models from scraping the public repos
FYI, there is the nuke option, which is generating endless nonsense pages as a form of “bot sink” [1] that ends the scraping relatively quickly, but IIRC that also tanks your search rankings, since it likely affects benign crawlers too – not something you’d necessarily want to happen to a new domain, unless you really need to protect server resources against aggressive hostile crawlers like SourceHut and so many others had to combat.
[1] https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrape...
Facepalm
LLMs are read only.
A developer would be the one using the LLM to evaluate page response text and scraping what they need.
LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods.
> LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods.
Being a bit needlessly pedantic here. I think it's well understood in these circles that such models are being driven by agents, which absolutely can crawl and scrape. The only thing a human needs to do is write rules for the agent to follow.
No, it allows people to dodge accountability.
It’s also not true.
An LLM cannot visit the internet at all.
There is no “agent” - that is a marketing term. It’s software and LLM that’s it.
3 replies →