Comment by motoboi
16 days ago
I insist on the text generator nature of the thing. It’s just that we built harnesses to activate on certain sequences of text.
Think of it as three people in a room. One (the director), says: you, with the red shirt, you are now a plane copilot. You, with the blue shirt, you are now the captain. You are about to take off from New York to Honolulu. Action.
Red: Fuel checked, captain. Want me to start the engines?
Blue: yes please, let’s follow the procedure. Engines at 80%.
Red: I’m executing: raise the levers to 80%
Director: levers raised.
Red: I’m executing: read engine stats meters.
Director: Stats read engine ok, thrust ok, accelerating to V0.
Now pretend the director, when heard “I’m executing: raise the levers to 80%”, instead of roleplaying, she actually issue a command to raise the engine levers of a plane to 80%. When she hears “I’m executing: read engine stats”, she actually get data from the plane and provide to the actor.
See how text generation for a role play can actually be used to act on the world?
In this mind experiment, the human is the blue shirt, Opus 4-6 is the red and Claude code is the director.
For context I've been an AI skeptic and am trying as hard as I can to continue to be.
I honestly think we've moved the goalposts. I'm saying this because, for the longest time, I thought that the chasm that AI couldn't cross was generality. By which I mean that you'd train a system, and it would work in that specific setting, and then you'd tweak just about anything at all, and it would fall over. Basically no AI technique truly generalized for the longest time. The new LLM techniques fall over in their own particular ways too, but it's increasingly difficult for even skeptics like me to deny that they provide meaningful value at least some of the time. And largely that's because they generalize so much better than previous systems (though not perfectly).
I've been playing with various models, as well as watching other team members do so. And I've seen Claude identify data races that have sat in our code base for nearly a decade, given a combination of a stack trace, access to the code, and a handful of human-written paragraphs about what the code is doing overall.
This isn't just a matter of adding harnesses. The fields of program analysis and program synthesis are old as dirt, and probably thousands of CS PhD have cut their teeth of trying to solve them. All of those systems had harnesses but they weren't nearly as effective, as general, and as broad as what current frontier LLMs can do. And on top of it all we're driving LLMs with inherently fuzzy natural language, which by definition requires high generality to avoid falling over simply due to the stochastic nature of how humans write prompts.
Now, I agree vehemently with the superficial point that LLMs are "just" text generators. But I think it's also increasingly missing the point given the empirical capabilities that the models clearly have. The real lesson of LLMs is not that they're somehow not text generators, it's that we as a species have somehow encoded intelligence into human language. And along with the new training regimes we've only just discovered how to unlock that.
> I thought that the chasm that AI couldn't cross was generality. By which I mean that you'd train a system, and it would work in that specific setting, and then you'd tweak just about anything at all, and it would fall over. Basically no AI technique truly generalized for the longest time.
That is still true though, transformers didn't cross into generality, instead it let the problem you can train the AI on be bigger.
So, instead of making a general AI, you make an AI that has trained on basically everything. As long as you move far enough away from everything that is on the internet or are close enough to something its overtrained on like memes it fails spectacularly, but of course most things exists in some from on the internet so it can do quite a lot.
The difference between this and a general intelligence like humans is that humans are trained primarily in jungles and woodlands thousands of years ago, yet we still can navigate modern society with those genes using our general ability to adapt to and understand new systems. An AI trained on jungles and woodlands survival wouldn't generalize to modern society like the human model does.
And this makes LLM fundamentally different to how human intelligence works still.
> And I've seen Claude identify data races that have sat in our code base for nearly a decade
how do you know that claude isn't just a very fast monkey with a very fast typewriter that throws things at you until one of them is true ?
Iteration is inherent to how computers work. There's nothing new or interesting about this.
The question is who prunes the space of possible answers. If the LLM spews things at you until it gets one right, then sure, you're in the scenario you outlined (and much less interesting). If it ultimately presents one option to the human, and that option is correct, then that's much more interesting. Even if the process is "monkeys on keyboards", does it matter?
There are plenty of optimization and verification algorithms that rely on "try things at random until you find one that works", but before modern LLMs no one accused these things of being monkeys on keyboards, despite it being literally what these things are.
2 replies →
For someone claiming to be an AI skeptic, your post here, and posts in your profile certainly seem to be at least partially AI written.
For someone claiming to be an AI skeptic, you certainly seem to post a lot of pro-AI comments.
Makes me wonder if this is an AI agent prompted to claim to be against AIs but then push AI agenda, much like the fake "walk away" movement.
I have an old account, you can read my history of comments and see if my style has changed. No need to take my word for it.
Tangential off topic, but reminds me of seeing so many defenses for Brexit that started with “I voted Remain but…”
Nowadays when I read “I am an AI skeptic but” I already know the comment is coming from someone that has just downed the kool aid.