Comment by asyncadventure
13 hours ago
This is incredibly useful for understanding the black box of LLM API calls. The real-time token tracking is game-changing for debugging why certain prompts are so expensive and optimizing context window usage. Having the markdown/JSON exports of every request makes it trivial to iterate on prompt engineering.
Curious - what pushed you toward a proxy vs adding observability/instrumentation in the code?
That is exactly the idea, later we can actually tap into the middle and optimize how the context is actually being used. Feels like the current anthropic tools like compact don't do a great job at it.