Comment by terminalshort
9 days ago
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
9 days ago
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
HEAD requests
Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?