Comment by mtrifonov

Two interpretations: either it's pure pattern-completion landing on the same trough, or whatever's underneath has a stable shape that the explanation tracks. Both are interesting. The "users don't understand the system" frame doesn't really pick between them.

Go watch an episode of COPS. Humans giving post-hoc explanations of their own behavior do the exact same thing.