Comment by nater5000

First, I actually had the author of this post as my distributed systems professor back in college. He's not just a super smart dude on a technical level, but he's probably the only CS professor I've ever had who seems to appreciate writing in itself (which may be evident from his blog). I think the combination of these two qualities puts him in a good spot to make the claims he is.

In terms of what he's presenting here, I'll say this: his line of logic highlighting the "dual-mind problem" of writing seems, to me, like the most compelling aspect of this argument. When I write something (especially to a specific audience, including an individual), I must practice cognitive empathy if I want what I write to be consumed properly and effectively. This is a cold way of putting it, but even in text message responses which span a few words towards family or a friend, I can put quite a bit of thought into how it will be received, how they will read it, interpret it, etc. My cadence in just texting, alone, can shift dramatically from one message to a next based on who I'm sending it to, what I'm trying to convey, etc.

All of that is fine and not hard to understand, but I suppose the interesting part is this: when I'm just trying to get information across (i.e., instructions, directions, etc.), my messages will resemble something written by an AI. It's not enjoyable to consume, but that's not the point. But as soon as I want to add a bit of "fun" to a message, the task I'm performing is completely different. It's no longer just an exchange of information, but it's an attempt to invoke specific feelings, visualizations, memories, etc., in the other person/people.

I do have a hard time imagining what it will take for AIs to be able to do THAT effectively. I think they're fine at conveying information, but I think people are already becoming very aware that conveying information, in itself, is not enough to be effective. These LLMs don't "care" to "entertain" you with what they're writing at you about. They're just spitting out the mathematically derived facts with, seemingly, no meaningful ability to invoke deeper thoughts in their audiences' minds. And that might not seem particularly important outside the context of writing fiction, etc., but I think it's actually pretty critical even in "dry" settings, like explaining code, because the thoughts, feelings, emotions, etc., invoked through reading IS the output of reading (even if it doesn't seem that way).

Perhaps AI will be able to do this more effectively if they're trained on direct brain signals or something. Like, train the AI not just to convey accurate information, but also encourage it to do so in a way which stimulates different areas of the brain with different intensities based on the premise that, doing so, is actually what people are getting out of that text.