Comment by pfdietz

The relevance is because the LLMs are storing information, not the explicit text, so we want to know how much actual information they need to store (this being an information theoretic argument). The representation in the parameters doesn't necessarily need 1 parameter per character, if the text is highly redundant.