Comment by JimmyRuska

This is great, love it!

Crawling sites to index the FAQ's and knowledge bases, into the vector search, isn't as intimidating as it sounds, at least on linux systems. Sometimes a thin wrapper function over plain old wget will get you 99% of the way

    wget -rnH -t 1 --waitretry=0 'https://{{domain}}' -P '{{domain}}'