Comment by mritchie712

8 days ago

We (https://www.definite.app/) have a use case I'd imagine is common for people building agents.

When a user works with our agent, they may end up with a large conversation thread (e.g. 200k+ tokens) with many SQL snippets, query results and database metadata (e.g. table and column info).

For example, if they ask "show me any companies that were heavily engaged at one point, but I haven't talked to in the last 90 days". This will pull in their schema (e.g. Hubspot), run a bunch of SQL, show them results, etc.

I want to allow the agent to search previous threads for answers so they don't need to have the conversation again, but chunking up the existing thread is non-trivial (e.g. you don't want to separate the question and answer, you may want to remove errors while retaining the correction, etc.).

Do you have any plans to support "auto chunking" for AI message[0] threads?

0 - e.g. https://platform.openai.com/docs/api-reference/messages/crea...

3 comments

mritchie712

Reply
snyy  7 days ago

> you may want to remove errors while retaining the correction

Double clicking on this, are these messages you’d want to drop from memory because they’re not part of the actual content (e.g. execution errors or warnings)? That kind of cleanup is something Chonkie can help with as a pre-processing step.

If you can share an example structure of your message threads, I can give more specific guidance. We've seen folks use Chonkie to chunk and embed AI chat threads — treating the resulting vector store as long-term memory. That way, you can RAG over past threads to recover context without redoing the conversation.

P.S. If HN isn’t ideal for going back and forth, feel free to send me an email at shreyash@chonkie.ai.

  • mritchie712  7 days ago

    > We've seen folks use Chonkie to chunk and embed AI chat threads

    yep, that's what we're looking for. We'll give it a shot!

    I think it's worth creating a guide for this use case. Seems like something many people would want to do and the input should be very similar across your users.