Comment by docfort
7 hours ago
I think the first goal of writing is to talk to yourself. Without an external, unchanging scratchpad, most people cannot keep their thoughts truly straight. It can feel otherwise, but trying to write something about any tricky topic even a few pages in length can take a long time to be self-consistent. Clarifying it takes even longer, especially when you intend to share it. Writing as thinking is definitely a thing. I believe it’s the trickiest part of understanding how such interesting things pop out of a language model trained to predict the next token and rewarded with pleasing sequences. Since it is not thinking, it proves that certain sentences (most?) can be generated thoughtlessly.
I like the term I've seen someone use for LLMs: a "thought partner." Truly it's the duck programming duck I've always wanted.
I say that LLM's are a pacifier. LLM output is to conscious output as a silicone nipple is to a breast.
Sure, at some level, there are some people who at least some times, cannot pecieve a difference. That does not prove that there is no difference or that the difference doesn't matter.
I think that LLM's are a broken mirror or maybe disco ball. What you get back is almost always a reflection of what you put in, just garbled through and with the reflections of so many other people. The best parts that I often get back are mostly just the stuff that I already put in it.
I think that's also where LLMs get the most dangerous. That ELIZA effect of telling you exactly what you told it but in a slightly rephrased way makes you think that you think that your ideas are genius. You've got someone "actively listening" and telling you whatever idea you just gave it was the best idea it ever heard.
That's not what you want from a serious writing partner. You want a writing partner to challenge you, to point out your mistakes and flaws, to make you better, to offer a truly different perspective to your own. You don't want a dark reflection from a "yes man" to spiral into.
Don't just focus on the model, half the turns in a session belong to the human, it's an instrument that depends on how well the user wields it. It does not need to produce the novel insights, just to support the user to have them.