Comment by HarHarVeryFunny

5 hours ago

No doubt the unreleased models are better at the things that LLMs are good at (coding, hacking, math) than what's been released, but there is still zero sign of their intelligence becoming broader and more general, and knowing how these models work and are trained there is no reason to expect them to become much more general unless changes are made.

The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.

3 comments

HarHarVeryFunny

ben_w 4 hours ago

> but there is still zero sign of their intelligence becoming broader and more general

Ahem, getting better at these things is the sign:

  It's hard to see how OpenAI or anyone else could provide this on-the-job training to an LLM even if they let it loose in a programming playground where it could generate the training dataset. How fast would the context fill with compiler/link errors, debugger output, program output etc ... once context was full you'd have to pre-train on that (very slow - months, expensive) before it could build on that experience. Days of human experience would take years to acquire. Maybe they could train it to write crud apps or some other low-hanging fruit, but it's hard to see this ever becoming the general purpose "AI programmer" some people think is around the corner. The programming challenges of any specialized domain or task would require training for that domain - it just doesn't scale. You really need each individual deployed instance of an LLM/AI to be able to learn itself - continuously and incrementally - to get the on-the-job training for any given use.

- you, two and a half years ago, https://news.ycombinator.com/item?id=39720601

HarHarVeryFunny 4 hours ago
Yes, I failed to anticipate environment/playground-based RL-training and how much difference that would make, although I think the broader point about limitations of lack of continual learning still stands, and I can't see these simulated environments ever become complex and/or varied enough to be any substitute for the need to learn on the job.
However, gotcha acknowledged, but my point was about the jaggedness of AI, and how huge the divide in capability is between the domains where RLVR works, primarily coding, hacking and math, and everything else. The success and scale of RLVR in areas such as coding only make this gap more noticeable.
- famouswaffles 3 hours ago
  
  This is what the frontier can do with a computer now.
  https://stillwet.art/
  Where are the verifiable rewards? It's simply not true that RLVR amenable domains are the only things LLMs are improving massively on. Even Vending Bench is improving.
  Intelligence is a jagged beast. That's just the nature of things. We evolved in certain environments and situations and sometimes there's a mismatch and it causes all sorts of wonky things. We just call them funny names like optical illusions and cognitive biases. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters (Watson Test).