← Back to context

Comment by driggs

18 hours ago

This site is going to be expensive when a web crawler hits it. A honey pot that burns tokens.

They’re caching the pages which have already been generated. You could go back and delete all references to pages which don’t exist yet. Basically turn it into a static website.

  • It seems like the site's algorithm is that every newly-generate page includes multiple links to not-yet-existing pages. So it doesn't matter that existing pages are cached, all the "leaf node" pages link to multiple uncached new pages.

    • I’m suggesting to turn that off and prune the links to pages which weren’t generated yet if cost becomes an issue.