Comment by quasarj

It's different to me from text written by another person, because it is awful in a way that even a bad human writer can't replicate without first performing a lobotomy on themself.

A great mental model here is that if you talk to many people, each one has their verbal ticks, their repeated phrases, the preferred idioms, but... you don't notice because you speak to each acquaintance only for a few minutes or hours at most at a week. If you really know someone, since you were both kids, or whatever, then you can often learn their particular style and even mimic it.

If you think of the the frontier AI models as people, there aren't hundreds or thousands of "intelligent human-like machines" that you're speaking to, there are like... two: whatever the best model is made by Open AI or Anthropic right now.

A programmer might be reading (or at least skimming) thousands of pages of text from just one (or at most two or three) models a week, week after week, for months now.

It's like the entire world is talking to two aliens that have stepped of their glittering spaceship. The aliens can hold millions of conversations at once, but they have only two personalities, so... that's what everyone is getting, and everyone has learned to recognise.

What makes things even more obvious is that unlike humans, the models don't "self adjust". On the contrary, they are inherently amnesiacs, and hence don't realise that they're awfully repetitive! A human using the same phrase fifty times in a row will dial it back a notch, but these dials are the weights of the models and are frozen for each released version.