Comment by skybrian

Although it doesn’t actually build the index, if AI crawlers really do want to save on crawling costs, couldn’t they share a common index? Seems like it’s up to them to build it.