Comment by mikewarot
2 months ago
Why don't people here on HN understand that LLMs never see ASCII or other raw characters as input?
Expecting spelling, rhyming, arithmetic or other character oriented responses will always yield disappointing results.
We do understand. We don't think that's okay. If a model cannot manage character level consideration, that's a serious flaw that's got potential to lead to an immeasurable number of failure states. "Duh, of course it can't count" is not the best look for a bot whose author tells us it's got PhD-level skill.
I do think it's "okay". After all, it's clear that fixing it would require a fundamentally different approach.
I just also think it's a reason to mock people who don't try to understand those limitations and get way ahead of themselves hyping up the technology.
The entire point of this exercise is to refute the claim that LLMs are a step towards AGI, even given "agency". And we should be happy that they aren't — because supposing that AGI is possible, the way that we currently treat LLMs shows that we as a species are nowhere near ready for the consequences of creating it.
Can you enumerate some of these "immeasurable number of failure states"? For me it starts and stops at "can't count letters in a word". That hardly seems catastrophic.
All I have to do is turn on thinking mode and the error goes away. https://chatgpt.com/share/6897e630-77f0-800c-a9bf-30d9c0e271...
So, if an AI can just spit out the cure for cancer, but spells some things wrong, it's not intelligent?
You think all PhD candidates have perfect spelling? I'd wager most of them re-read their dissertation and edit it, over and over, a process that most LLMs don't have the luxury of doing.
We'd have to give up all the efficiency of tokenizing, re-train a model (a much less optimum model) for at least twice as long to get anywhere near the same results for one that just spits out ASCII.
"LLMs are cool tools with clear limitations" is not the narrative being pushed by the bosses and boosters. "LLMs are literal magic that will replace large portions of the workforce and be a bigger revolution than fire" is what they are saying.
Because the damn things are marketed under the word "intelligence". That word used to mean something.
What did it used to mean? I was under the impression that it has always be a little vague.
Sure. Language is squishy, and psychometrics is hard. Nevertheless...
"Intelligence" refers to a basket of different capabilities. Some of them are borderline cases that are hard to define. The stuff that GPT-5 failed to do here is not.
Things like knowing what a question means, knowing what you know and don't, counting a single digit number of items, or replying with humility if you get stuck -- these are fairly central examples of what a very, very basic intelligence should entail.
It's an umwelt problem. Bats think we're idiots because we don't hear ultrasonic sound, and thus can't echolocate. And we call the LLMs idiots because they consume tokenized inputs, and don't have access to the raw character stream.
If you open your mind up too far, your brain will fall out.
LLMs are not intelligence. There's not some groovy sense in which we and they are both intelligent, just thinking on a different wavelength. Machines do not think.
We are inundated with this anthropomorphic chatter about them, and need to constantly deprogram ourselves.
Do bat's know what senses humans have? Or have the concept of what a human is compared to other organisms or moving objects? What is this analogy?
1 reply →
> we call the LLMs
"Dangerous", because they lead into thinking they do the advanced of what they don't do basically.
Pretty sure bats don’t think about human intelligence at all.
And which other objectual ideas cannot they instance? Their task is to check, for all important mental activities - world simulation, "telling yourself reliable stories: that is what intelligence is" (Prof. Patrick Winston).
The only issue is they shouldn't call it PHD level intelligence when they can't do simple task like this.