Comment by mlsu

> if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it

There are sociological reasons why this happens less with humans:

1. You cycle your dumb repetitive jokes with everyone you meet, so nobody hears it twice

2. Those who know you well will notice when you're just repeating ("dad jokes")

3. As a person's idiosyncrasies are beginning to wear on their social circles, they will be getting small clues to stop saying those things. Agents don't get these social between-the-lines cues to stop a certain behavior, they endlessly repeat. Perhaps between model version releases, frontier labs can harvest the web and ask "What Claudisms do people mention negatively?" but I don't think they do that yet.

I think that there is a sort of mechanism by which, when a human speaks to you, you can "mirror" their internal mental state, and the quality of writing often corresponds to how much you get pleasure or information or whatever your goal is from that mental state. The important part is that the words themselves are just pointers to the state. So, the person speaking, if they are skillful, gives enough words, and enough variety, that you can start to produce a state yourself which resembles theirs.

This is the thing that LLM writing doesn't really do. Since there isn't a mental state, there's nothing to mirror anyway, and somehow you can detect the absence of it even though it's hard to put any of the machinery into words. Corporate speech also fails to activate this machinery in the same way, but LLMs seem to do it more egregiously, probably because the corporate speech was at least compiled by a human---even if it is not the thoughts of an individual, it is the "thoughts" of an "entity", the abstract corporation, which the writer was speaking for, and so you can still wrap your mind around the fact that it is communicating with you.

> I am curious why LLM writing has such an uncanny valley feel to it.

Because they are HEAVILY trained to give addictive responses.

They don't want to just answer your question. They want to sycophantically make you feel like a genius for being smart enough to use them.

One reason is that one person’s idiosyncracies are limited in scope, but LLM-produced text is now everywhere. Also, filler words and mannerisms in speech we’re quite good at filtering out, but in written text the stand out much more.

It’s the repetitiveness of style, the attempt to make everything seem as impactful as possible, the use of short sentences (too much Hemingway in the training data?), and obvious patterns like “it’s not this, it’s that” and several others.

Real human writing doesn’t follow such strict rules. When the same small set of rules is applied over and over throughout a text, it becomes obviously strange and machine-like.

I had pretty good luck recently by giving it a writing guide about word choice, sentence structure, paragraph structure, and overall doc structure. I basically ask it to read the guide and revise a couple of times before I engage with its writing. Ymmv.

Next RL phase is to connect electrodes to human brain when reading generated output to reduce frustration signal.

I don't think its an inherent quality - a lot of older models had a much more natural feel to them.

I guess it has to do with the low temperature (low randomness in final word choce - something all vendors seem to have converged on for some reason) which does make them less likely to skiz out but makes the writing feel dry and samey. It's like repeating a list of dice rolls, and replaying them - there's no inherent pattern in the input, but there sure is in the output.

I suspect because the English colloquial text training data it had access to was early 2000's message boards and social media. Therefore it uses "honestly" and other expressions that took hold in the late 90s and early 2000s.

Think of it as a mad lib, it’s populating a template, and seeing the same template filled over and over gets tiring.

> I am curious why LLM writing has such an uncanny valley feel to it.

Because it's trained to talk like a marketing committee.

I always assumed it was just a form of fingerprinting or the result thereof.

Very surprised no one has this answer: because it is fundamentally not a human being.

I’d guess it’s because that’s exactly how its System Template was written to do

[dead]

Might be related to their fingerprinting of llm output they said earlier in the week.

I agree that it’s better at writing than a 50%-ile human, but it’s worse at communicating through writing than most humans.

Even an average human writer can communicate details much more succinctly and directly than an LLM

No, it does not.

My personal bugbear is its usage of "grain" where normally you'd use "granularity", if at all.

Belt and braces all the way down. They did their post training in British context?

Provenance

I've been noticing the same thing since 5.4- it starts with "Yes" yet I haven't asked a question. I think it might be related to the reasoning, like it's answering its own questions?

i am having fun imagining an agent existentially freaking out while trying to parse this directive since inference is the entirety of their world

That's exactly right. And as is well understood, a good pseudorandom generator, despite being fully deterministic, is extremely hard to distinguish from randomness, unless you have the algorithm and key (internal state). Quite smart, really.

The watermarking is independent of the model. The model itself has the probability weights to determine the next token. The watermark is similar to things like temperature and top_p/top_k in that the watermark adjusts the probabilities in a deterministic way that changes over time to hide tells from word choices.

I think I understand what you are trying to convey but I fear you've made the same mistake again, this time with "soul" instead of "intelligence."

I think what you are getting at is that they are deterministic automata. They are machines. We have introduced randomness to add variation but it is an artificial randomness that simply perturbs the path traversed.

When we choose words it isn't because of a token distribution, nor because we rolled a die. We choose words because we feel a certain way, the external world, our body and senses are all connected as one system. These machines don't experience moods or get tired or feel better after a good night's sleep. They don't know their audience, we're all the same to them. We have no personal relationship nor can we establish one, as presenting some arbitrary background is not the same thing as a fluid, evolving relationship that accumulates through experience over time. There are no scars or fond memories.

If these things can truly be intelligent, to abuse your use of the word, then at least we are quite far from holding them correctly.

    Will Claude will act as a therapist or produce 
    value or produce a work of art? No. It cannot, 
    because it does not have a soul. 

I tentatively agree, although I'm only tentative because I don't think it's an interesting question.

Here's what I do think is interesting. You!

I mean... yes, you, too but not you specifically. The plural "you" that the english language lacks.

And so here's what I think is the actual interesting question. Might AI help you create art? Or be a therapist? Or something else interesting and worthwhile?

Maybe AI won't write the next great guitar solo. I'm pretty sure it won't. But might it help you learn to play guitar? Help you fix your broken guitar amp? Help you understand some tricky parts of guitar playing? Help you work through some tricky tabulature where you can't tell if you're playing it wrong or if the tab is just bad?

I don't know. But that's my angle for finding any of this interesting.

“It is not sharing anything, because it is not capable of sharing, because it does not have a soul”

soul.md though, just saying