Comment by r3trohack3r
1 day ago
I’ve struggled with this line of thought. In a strict sense, it’s true; an AI model doesn’t actually know why it fired the way it did. But I don’t know how practically useful that observation is because, using that strict definition of “know”, humans also do not know why they do what they do.
If I asked you why you posted this comment, I suspect you’d be able to articulate a good reason why from your memories. But the firing of your subconscious and the entirety of your lived experience that molded the state in your brain that lead to you pressing that reply button and writing those exact words are inaccessible to you.
It reminds me of the timeline exercise in SRE work. I always start with a timeline and plunk pagers, metrics, slack messages, etc onto that timeline first. Then I go to individual people to get their understanding of the incident and get that into the timeline. People’s understanding of the incident and why they did what they did rarely matches the timeline.
When I ask an LLM to review its reasoning over prior conversation turns, it has access to its tool call results, web searches, and reasoning traces that aren't visible to me in the UI.
There's definitely value in the "meta-cognition" that an LLM can do even if it is blind to the underlying weights and layers of the model itself.