Slacker News Slacker News logo featuring a lazy sloth with a folded newspaper hat
  • top
  • new
  • show
  • ask
  • jobs
Library
← Back to context

Comment by jsheard

6 months ago

The sitemap protocol does have some features to help avoid unnecessary crawling, you can specify the last time each page was modified and roughly how frequently they're expected to be modified in the future so that crawlers can skip pulling them again when nothing has meaningfully changed.

0 comments

jsheard

Reply

No comments yet

Contribute on Hacker News ↗

Slacker News

Product

  • API Reference
  • Hacker News RSS
  • Source on GitHub

Community

  • Support Ukraine
  • Equal Justice Initiative
  • GiveWell Charities