Comment by NitpickLawyer
2 months ago
> and that LLMs have basically reached a plateau
This is the new stochastic parrots meme. Just a few hours ago there was a story on the front page where an LLM based "agent" was given 3 tools to search e-mails and the simple task "find my brother's kid's name", and it was able to systematically work the problem, search, refine the search, and infer the correct name from an e-mail not mentioning anything other than "X's favourite foods" with a link to a youtube video. Come on!
That's not to mention things like alphaevolve, microsoft's agentic test demo w/ copilot running a browser, exploring functionality and writing playright tests, and all the advances in coding.
And we also have a showcase from a day ago [1] of these magical autonomous AI agents failing miserably in the PRs unleashed on the dotnet codebase, where it kept reiterating it fixed tests it wrote that failed without fixing them. Oh, and multiple blatant failures that happened live on stage [2], with the speaker trying to sweep the failures under the rug on some of the simplest code imaginable.
But sure, it managed to find a name buried in some emails after being told to... Search through emails. Wow. Such magic
[1] https://news.ycombinator.com/item?id=44050152 [2] https://news.ycombinator.com/item?id=44056530
Is this something that the models from 4 months ago were not able to do?
For a fair definition of able, yes. Those models had no ability to engage in a search of emails.
What’s special about it is that it required no handholding; that is new.
Is this because the models improved, or the tooling around models improved (both visible and not visible to the end user).
My impression is that the base models have not improved dramatically in the last 6 months and incremental improvements in those models is becoming extremely expensive.
3 replies →
The LLMs have reached a plateau. Successive generations will be marginally better.
We're watching innovation move into the use and application of LLMs.
Innovation and better application of a relatively fixed amount of intelligence got us from wood spears to the moon.
So even if the plateau is real (which I doubt given the pace of new releases and things like AlphaEvolve) and we'd only expect small fundamental improvements some "better applications" could still mean a lot of untapped potential.
The core models have plateaued. MoE and CoT are use of LLMs. Agents are applications of LLMs. It's hard to say how far novel uses and applications will take us, but the fiery explosion at the core has turned into a smolder.
We'll continue to see incremental improvements as training sets, weights, size, and compute improve. But they're incremental.