Comment by pjm331
6 hours ago
Yeah i have a similar setup at the harness level - a “devlog” at the end of every session about the experience developing, what could have been better, what was confusing
And also files issues for blockers
I’ve absolutely caught things and made improvements just from skimming them occasionally - they are particularly useful when you get a PR that makes you scratch your head
But I’m definitely not taking full advantage of all the feedback coming in yet
I have to imagine parsing signal from noise there is a massive challenge when it’s other agents that are using your MCP and not just your own
You won’t get grounded results by asking for feedback. The LLM doesn’t know its own internal state so while you may get a plausible and contextually significant answer, it isn’t going to be an accurate one.
Why are you asking an LLM what could have been better or was confusing? It literally has no idea.
True. But it has no idea that it has no idea, so it might be able to look back at the session trace and pattern match its way to actionable feedback?
If they try something and it doesn't work and they then try something else and it DOES work that's concrete feedback.
Are you saying the machine can tell you what confused them?
they're surprising good at this. The next tokens after a confusing string of tokens when asked to explain the confusing parts are pretty accurate!