Comment by Borealid
13 days ago
> No refusal fires, no warning appears — the probability just moves
I don't really understand why this type of pattern occurs, where the later words in a sentence don't properly connect to the earlier ones in AI-generated text.
"The probability just moves" should, in fluent English, be something like "the model just selects a different word". And "no warning appears" shouldn't be in the sentence at all, as it adds nothing that couldn't be better said by "the model neither refuses nor equivocates".
I wish I better understood how ingesting and averaging large amounts of text produced such a success in building syntactically-valid clauses and such a failure in building semantically-sensible ones. These LLM sentences are junk food, high in caloric word count and devoid of the nutrition of meaning.
Surely I cannot be the only one who finds some degree of humor in a bunch of nerds being put off by the first gen of "real" AI being much more like a charismatic extroverted socialite than a strictly logical monotone robot.
In a way, it’s a simulacrum of a saas b2b marketing consultant because that’s like half the internet’s personality
It's funny but I'm on HN so I can't resist pointing out the joke doesn't math TFA, their argument is that the underlying internet distribution is trained away, not retained.
2 replies →
Not particularly charismatic, just looks a lot like the worst kind of yapping wannabe.
Charismatic extroverted socialites dont talk that way. They do not make mistakes like that.
That's a great description of the boundary between logical deduction NLP and bullshitting NLP.
I still have hope for the former. In fact, I think I might have figured out how to make it happen. Of course, if it works, the result won't be stubborn and monotone..
The axis running from repulsive to charismatic, the axis running from hollow to richly meaningful, and the axis running from emotional to observable are not parallel to each other. A work of communication can be at any point along each of those three independent scales. You are implying they are all the same thing.
I hate it because typically that style of writing was when someone cared about what they were writing.
While it wasn't a great signal it was a decent one since no one bothered with garbage posts to phrase it nicely like that.
Now any old prompt can become what at first glance is something someone spent time thinking about even if it is just slop made to look nice.
This doesn't mean anything AI is bad, just that if AI made it look nice that isn't inductive of care in the underlying content.
> I hate it because typically that style of writing was when someone cared about what they were writing.
I dont understand these takes. The opposite is true - humans good at writing who care about writing never produced these kind of texts.
People who dont care about writing, but need to crank up a lot of words would occasionally produce writing like that. Human slop existed before ai, but it was not the thing produced by people who write well and care.
1 reply →
I always felt like humans that were good at writing that way were often doing exactly what the LLM is doing. Making it sound good so that the human reader would draw all those same inferences.
You've just had it exposed that it is easy to write very good-sounding slop. I really don't think the LLMs invented that.
2 replies →
hahaha amazing
[dead]
[flagged]
I doubt you've ever thrown a drink in anyone's face, and I hope I'm right. This kind of thing isn't appropriate for HN.
5 replies →
Please, I'm just a self aware nerd.
8 replies →
It's really simple. RL on human evaluators selects for this kind of 'rhetorical structure with nonsensical content'.
Train on a thousand tasks with a thousand human evaluators and you have trained a thousand times on 'affect a human' and only once on any given task.
By necessity, you will get outputs that make lots of sense in the space of general patterns that affect people, but don't in the object level reality of what's actually being said. The model has been trained 1000x more on the former.
Put another way: the framing is hyper-sensical while the content is gibberish.
This is a very reliable tell for AI generated content (well, highly RL'd content, anyway).
<https://en.wikipedia.org/wiki/Supernormal_stimulus>
Neural networks are universal approximators. The function being approximated in an LLM is the mental process required to write like a human. Thinking of it as an averaging devoid of meaning is not really correct.
> The function being approximated in an LLM is the mental process required to write like a human.
Quibble: That can be read as "it's approximating the process humans use to make data", which I think is a bit reaching compared to "it's approximating the data humans emit... using its own process which might turn out to be extremely alien."
Good point.
Then again, whatever process we're using, evolution found it in the solution space, using even more constrained search than we did, in that every intermediary step had to be non-negative on the margin in terms of organism survival. Yet find it did, so one has to wonder: if it was so easy for a blind, greedy optimizer to random-walk into human intelligence, perhaps there are attractors in this solution space. If that's the case, then LLMs may be approximating more than merely outcomes - perhaps the process, too.
11 replies →
I don't think of it as "devoid of meaning". It's just curious to me that minimizing a loss function somehow results in sentences that look right but still... aren't. Like the one I quoted.
A human in school might try to minimise the difference between their grades and the best possible grades. If they're a poor student they might start using more advanced vocabulary, sometimes with an inadequate grasp of when it is appropriate.
Because the training process of LLMs is so thoroughly mathematicalised, it feels very different from the world of humans, but in many ways it's just a model of the same kinds of things we're used to.
> Thinking of it as an averaging devoid of meaning is not really correct.
To me, this sentence contradicts the sentence before it. What would you say neural networks are then? Conscious?
They are a mathematical function that has been found during a search that was designed to find functions that produce the same output as conscious beings writing meaningful works.
2 replies →
>I wish I better understood how ingesting and averaging large amounts of text produced such a success in building syntactically-valid clauses
I wonder if these LLMs are succumbing to the precocious teacher's pet syndrome, where a student gets rewarded for using big words and certain styles that they think will get better grades (rather than working on trying to convey ideas better, etc).
This is more or less what happens. These models are tuned with reinforcement learning from human feedback (RLHF). Humans give them feedback that this type of language is good.
The notorious "it's not X, it's Y" pattern is somewhat rare from actual humans, but it's catnip for the humans providing the feedback.
> I wish I better understood how ingesting and averaging large amounts of text produced such a success in building syntactically-valid clauses and such a failure in building semantically-sensible ones. These LLM sentences are junk food, high in caloric word count and devoid of the nutrition of meaning.
I suspect that's because human language is selected for meaningful phrases due to being part of a process that's related to predicting future states of the world. Though it might be interesting to compare domains of thought with less precision to those like engineering where making accurate predictions is necessary.
> I don't really understand why this type of pattern occurs, where the later words in a sentence don't properly connect to the earlier ones in AI-generated text.
Because AI is not intelligent, it doesn't "know" what it previously output even a token ago. People keep saying this, but it's quite literally fancy autocorrect. LLMs traverse optimized paths along multi-dimensional manifolds and trick our wrinkly grey matter into thinking we're being talked to. Super powerful and very fun to work with, but assuming a ghost in the shell would be illusory.
> Because AI is not intelligent, it doesn't "know" what it previously output even a token ago.
Of course it knows what it output a token ago, that's the whole point of attention and the whole basis of the quadratic curse.
> Of course it knows what it output a token ago...
It doesn't know anything. It has a bunch of weights that were updated by the previous stuff in the token stream. At least our brains, whatever they do, certainly don't function like that.
12 replies →
If all the training data contains semantically-meaningful sentences it should be possible to build a network optimized for generating semantically-meaningful sentence primarily/only.
But we don't appear to have entirely done that yet. It's just curious to me that the linguistic structure is there while the "intelligence", as you call it, is not.
> If all the training data contains semantically-meaningful sentences it should be possible to build a network optimized for generating semantically-meaningful sentence primarily/only.
Not necessarily. You can check this yourself by building a very simple Markov Chain. You can then use the weights generated by feeding it Moby Dick or whatever, and this gap will be way more obvious. Generated sentences will be "grammatically" correct, but semantically often very wrong. Clearly LLMs are way more sophisticated than a home-made Markov Chain, but I think it's helpful to see the probabilities kind of "leak through."
7 replies →
Sentences only have semantic meaning because you have experiences that they map to. The LLM isn't training on the experiences, just the characters. At least, that seems about right to me.
4 replies →
Why would that be curious? The network is trained on the linguistic structure, not the "intelligence."
It's a difficult thing to produce a body of text that conveys a particular meaning, even for simple concepts, especially if you're seeking brevity. The editing process is not in the training set, so we're hoping to replicate it simply by looking at the final output.
How effectively do you suppose model training differentiates between low quality verbiage and high quality prose? I think that itself would be a fascinatingly hard problem that, if we could train a machine to do, would deliver plenty of value simply as a classifier.
I’m not up with what all the training data is exactly.
If it contains the entire corpus of recorded human knowledge…
And most of everything is shit…
1 reply →
Because AI is not intelligent, it doesn't "know" what it previously output even a token ago.
You have no idea what you're talking about. I mean, literally no idea, if you truly believe that.
That's only true if you consider the process the LLM is undergoing to be a faithful replica of the processes in the brain, right?
1 reply →