Comment by SwellJoe
2 hours ago
"Every run is traceable
Everything the model sees is recorded in an append-only session log: system prompts, reasoning, tool calls and results, subagent scheduling, and every context injection. In the Trajectory view, you can inspect these records by source. Resume, fork, search, and replay all operate on the same event stream."
That's a killer feature, IMHO, and one that US models won't allow you to do, as their traces are encrypted, obfuscated, etc. and have to be extracted via various workarounds (that violate the terms of service).
If you want to be able to improve your tools that work with models, you have to be able to assess what the models think is happening, how they think about and interact with the data you give them. And, the US models won't let you see that.
Agreed that it is a killer feature. US models obfuscate the COT (to A. make it look better and B. combat distillation) but > and the raw trace is fairly hard to reason about > but I still think this kind of feature is a big step in the right direction.
That is precisely how the dreamcoder [0] ai agent I built also works.
It has an event sourced architecture in SQLite and it resolves queries using recursive CTEs (and sneaky projections to speed things up) to deliver exactly that. Identical, stable message chains to AI and complete introspection.
Bonus points include a constraint-satisfaction solver for the tiling window manager so windows never shrink too small to read. And many other keyboard-friendly features.
[0] https://www.dreamcoder.ai/ [1] https://www.dreamcoder.ai/assets/graph.webp
How is that different from what Pi already does?
[delayed]
For the US models, you can look at the rewritten CoTs or just ask them what they think is happening.
Asking them what they think is happening is actually not reliable, though? They don't always know how they came to a conclusion after the fact. It is probable that it's roughly similar to the path they took to get there, since it's the same weights, but it's not certain. And, if you make it standard practice to always collect that data (e.g. if you have an automated tool to ask the model to explain itself after every action to log it), it seems like you might find yourself being blocked for violating terms of service. It looks like "distilling".
In short, there are workarounds, but they're not guaranteed to work forever and they're likely to bump into terms of service.