Comment by avereveard

I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.