Comment by boredatoms
8 hours ago
Can I suggest putting some text in the page that tells the bot what the more efficient download method is?
8 hours ago
Can I suggest putting some text in the page that tells the bot what the more efficient download method is?
What makes you think that would accomplish anything?
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.