Comment by Plont

That's exactly the problem though; humans will anthropomorphize everything from a roomba to a rock as it is. We're weak to this. We evolved in environments where looking for and recognizing real animals and humans was vital for survival, and false positives were relatively harmless. (Or potentially sometimes even helpful, like in the case of stuffed animals providing comfort to children.)

Now we have this software that's specifically designed to mimic humans, and mistaking it for real intelligence or consciousness can easily be disastrous. It's very important that we not anthropomorphize it, but we are catastrophically bad at NOT doing that.

Even our language has had a lot of computer anthropomorphism baked into it ("my phone died!", "this laptop is fussy", "the computer is sleeping", "it's thinking"), and it's not easy to excise that routine anthropomorphism from the way we talk about LLMs.

I don't want an LLM to write as if it were a person because it's definitely easier and more reliable to cut that problem off at the root, as much as possible, rather than to just try to willpower my way out of my human tendency to anthropomorphize inanimate objects.

I'll grant that LLMs talk like this because they're trained on human writing. It may not be possible to get them to not do that. But if it can't be fixed, it's just another thing to put on the "reasons this is all an incredibly stupid idea" pile.