Comment by ThrustVectoring
11 hours ago
Human writers also use specific constructions for definite (though not always consciously known) reasons, and mapping those correlations onto writing models is only getting done on accident. This is a big part of why LLMs doing pointless negations (it's not X, it's Y) is so jarring: people use that construction in order to preempt an expected misconception, whereas LLMs try to use linear algebra over the learning corpus to generate it in appropriate quantities and places according to RLHF/fine-tuning/etc.
In other words, it isn't just verbiage that has a "preferred" quantity, but also higher-level language features, and the appropriate contexts for those features feel like the sort of thing that will be extraordinarily difficult to replicate without a theory of mind.
edit after coming back and re-reading: the "in other words," and rephrase is because I'm trying to explain something complex enough that I'm not even sure I fully understand it, so I try typing out multiple explanations and leave the ones that make the most sense to me. This might be idiosyncratic to my writing and not a significant feature of training corpuses, but even if it wasn't it's a second example of the sort of thing I'd suspect LLMs to be bad at using appropriately.
Another LLM weirdness is to explain what code is not doing. My theory: such negations were somehow encouraged in training because they are very valuable where used properly (and sparely) by humans. "Not doing the obvious thing due to xyz happening two modules over" is a very useful comment. "Not using bubblesort because quicksort is faster" is garbage.
I think a lot of the time it's prompt leakage. Forgive the personification, but it's like they don't have a mental model of the reader of the thing that they're authoring being separate from the person who's prompting them to author it, so it often seems like the main conversation kind of spills into the prose.
Not to pick on GenHTTP Lambda, especially because I think it's someone's fun side project, but I was just trying to read though their docs and I was finding it super difficult. I kept getting distracted thinking "why the hell did it write it _that_ way?"
Like, the first sentence of "The control center" section [1] is the exact sort of weird negation that you're describing:
> The editor link opens a control center rather than a text box: most of the code here is written by agents, so the first thing on the screen is how your lambda is doing.
I can't help but think it generated the first round of the page with a text box to paste the code into, got told "no, I want a control center," and then this ended up in the docs.
I also think the phrasing of that sentence (and many others in those docs) is just super weird and unnatural - e.g. "so the first thing on the screen is how your lambda is doing." It also feels like it mirrors a prompt - like, "no, it should be a control center, not a text box, and the first thing the user should see is how their lambda is doing." But in docs I'd never expect it to be written "the status of your lambda" or "the health of your lambda." And what does the code being written by agents have to do with it?
1: https://genhttp.dev/en/docs#editor
Lambda is my side project and I 100% agree. Updating documentation with Claude does not work, unless you spend so much effort that you could have written a perfect version yourself. Opus 5.5 got better at it, but it's still a long way to go. I personally cannot stand the PR descriptions generated by Claude. Three pages of text for just a simple change ...
1000% this. I encounter this all the time (with Claude) and it drives me nuts!