Comment by smath

Here is a solution to this problem I think: make an LLM. Summarize everything. If there is fluff then it should get dropped? Basically we only care about the relevant information content, regardless of the number of characters used - so we need a compressed representation