Comment by wahern

The context of the whole essay was more about the way we have a tendency to confuse models with reality, in particular the way we build human-like machines based on contemporary theories about how humans work, then later fall into the assumption the theories were accurate simply because the machines are indeed useful. It's a pernicious kind of self-reinforcing confirmation bias. And it applies at least as much to AI, and LLMs specifically, as any earlier example. Whether or not one believes LLMs strongly parallel certain human mental processes, or even if in fact they do, surely we must admit there's much more undiscovered complexity and phenomena, and the pursuit of discovering those phenomena is going to be hampered to some extent by our investment in understanding them in terms of these otherwise successful tools we've built.