Comment by wmchen

Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future.

The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister.

I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1]

[1] "TODO store new documents in both indexes while running reindex to guarantee not losing any data." @ https://github.com/asciimoo/hister/blob/master/server/indexe...