Comment by mofeien
1 hour ago
When, in the past three years, has model progress seemed to decelerate to you, indicating some limit?
The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: https://aistatement.com/work/statement-on-ai-extinction-risk
They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
It's pretty telling that even with RL post-training the big labs have essentially made little progress on the hallucination rate of models. The issue is fundamental to the current paradigm, contrary to humans.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
Full speed ahead like an idiot savant trying a thousand different possibilities, though half of which are without basis in reality.
[1]:https://artificialanalysis.ai/evaluations/omniscience#omnisc...
Hallucination rate is a highly nonlinear metric relative to other model success metrics. A similar phenomenon to what is going on here: https://arxiv.org/abs/2304.15004 . This does not mean progress has stalled.
Anyone that thinks that the hallucination rate is 59% has not actually used these models on a real project.
Hallucinations come up when asking knowledge bases questions, such as "In React’s Canary Fragment refs API, which FragmentInstance method returns a flat array of DOMRect objects for all children?" Or "Over what years did Roubini and Sachs examine 15 OECD countries when assessing trends in tax-to-GDP ratios?" While to me those may seem hyper-specific and unlikely to come up in a real context, students will absolutely ask questions similar to these.
1 reply →
Are you suggesting that it's higher or lower?
IME on real projects you do need to be very careful with prompts about topics that are less likely be common in the training set.
I don't like the term 'hallucination' to be honest, not because it anthropomorphizes, but because it lacks a formal definition in the context of machine learning.
Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
https://arxiv.org/abs/2404.01019
Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
But then all the copy rights holders can simultaneously sue them.
And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
It's telling that the companies prefer regulation over source-aware training.
You clearly don't understand the underlying fundamentals of LLMs, harnesses, agents.
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
> They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
So, they're either liars or homicidally reckless.
Unfortunately if the frontier labs that care about safety stop or slow down unilaterally, that doesn’t make the problem go away. It makes it worse when labs that do not care at all about safety, or deny that it is even a problem, are leading the way. That’s why there’s needs to be some form of international regulation.
Incidentally this regulation won’t really touch the US providers - at least it sounds like that’s the plan if you listen to current US admin
Why not both?
My guess is its a combination of both. Either way I'm not a fan.
> either liars
They're CEOs of tech companies with insane valuations
> or homicidally reckless
They're CEOs of bleeding edge tech companies with huge capital and military applications
>So, they're either liars or homicidally reckless.
Yes!
So either they’re full of shit or we need to stop them by any means necessary.
Or there's a middle path where you cure most death and disease by ensuring governments and society responsibly regulates superintelligence.
Actually... nah. Why even try? Trendy cynical hot takes on social media are more fun!
I don’t believe them. I don’t believe that they are truly concerned about much beyond their own self interests.
3 replies →
“Curing” death would be the most dystopian devastating outcome i can think of
the ceiling of abilities seem to be growing steadily but the floor of errors seems to not change. New models can do more and more but still fail at seemingly (to human) simple tasks