Comment by tossandthrow

If you think about chunking as "take x characters" then using LLMs is a poor idea.

But syntactic chunking also works really poorly for any serious application as you loose basically all context.

Semantic chunking, however, is a task you absolutely would use LLMs for.