When is the last time you used a frontier model? Any chance you work someplace that forces you to use last year's model (or any American open weight model)?
I use them every day, in systems programming. They produce a solution, but it’s often several orders of magnitude away from an optimal one in various dimensions. That’s still quite useful, but for some problems that doesn’t cut it.
To reach a more optimal solution, significant human steering is reqiured for every tiny decision. It’s the same as how early models needed hand holding on the basics, playing out again in the finer details. Human experts need to inject “eureka moments” from the outside.
And no feedback loop that I’ve tried has been able to conjure an understanding of these finer details that, as far as I can tell, just aren’t represented in the weights.
Modeling the basics of software engineering was easy, since things like writing code that passes some tests is easy to RL. But for anything not easily RL-able, we’ve basically ran out of human generated tokens to distill from.
Without some breakthrough in generalization, I remain unconvinced that we’ll be able to model the finer details of software engineering by looping the existing training corpus back into itself. Garbage in, garbage out.
When is the last time you used a frontier model? Any chance you work someplace that forces you to use last year's model (or any American open weight model)?
I use them every day, in systems programming. They produce a solution, but it’s often several orders of magnitude away from an optimal one in various dimensions. That’s still quite useful, but for some problems that doesn’t cut it.
To reach a more optimal solution, significant human steering is reqiured for every tiny decision. It’s the same as how early models needed hand holding on the basics, playing out again in the finer details. Human experts need to inject “eureka moments” from the outside.
And no feedback loop that I’ve tried has been able to conjure an understanding of these finer details that, as far as I can tell, just aren’t represented in the weights.
Modeling the basics of software engineering was easy, since things like writing code that passes some tests is easy to RL. But for anything not easily RL-able, we’ve basically ran out of human generated tokens to distill from.
Without some breakthrough in generalization, I remain unconvinced that we’ll be able to model the finer details of software engineering by looping the existing training corpus back into itself. Garbage in, garbage out.
That's my experience as well. But: to be fair, they can be super good at debugging.