Comment by jsheard
19 days ago
The sitemap protocol does have some features to help avoid unnecessary crawling, you can specify the last time each page was modified and roughly how frequently they're expected to be modified in the future so that crawlers can skip pulling them again when nothing has meaningfully changed.
No comments yet
Contribute on Hacker News ↗