Comment by JimmyRuska

3 years ago

This is great, love it!

Crawling sites to index the FAQ's and knowledge bases, into the vector search, isn't as intimidating as it sounds, at least on linux systems. Sometimes a thin wrapper function over plain old wget will get you 99% of the way

    wget -rnH -t 1 --waitretry=0 'https://{{domain}}' -P '{{domain}}'

0 comments

JimmyRuska

Reply