Comment by jsheard

20 days ago

The sitemap protocol does have some features to help avoid unnecessary crawling, you can specify the last time each page was modified and roughly how frequently they're expected to be modified in the future so that crawlers can skip pulling them again when nothing has meaningfully changed.