Comment by brookst
13 hours ago
I just use a suite of tests to enforce the mechanical standards (dry, encapsulation, etc) and arch documents for shape + reviews of new work against the two. I’m not seeing any significant drift.
13 hours ago
I just use a suite of tests to enforce the mechanical standards (dry, encapsulation, etc) and arch documents for shape + reviews of new work against the two. I’m not seeing any significant drift.
Yea, this is one thing that gets me. Vibe coding and freestyle prompting gets wildly different results. Starting from a set of specs and expectations routinely gets me reviewable and we'll tested code. Sure, running the full test suite alot slows things down and uses more tokens, but when I hit a five hour or weekly window, I just roll up my sleeves and start working without the model because the code is layed out the way I expected and is designed to fit my meatbag workflow and pace.
You must not be looking very closely yourself then.
When is the last time you used a frontier model? Any chance you work someplace that forces you to use last year's model (or any American open weight model)?
I use them every day, in systems programming. They produce a solution, but it’s often several orders of magnitude away from an optimal one in various dimensions. That’s still quite useful, but for some problems that doesn’t cut it.
To reach a more optimal solution, significant human steering is reqiured for every tiny decision. It’s the same as how early models needed hand holding on the basics, playing out again in the finer details. Human experts need to inject “eureka moments” from the outside.
And no feedback loop that I’ve tried has been able to conjure an understanding of these finer details that, as far as I can tell, just aren’t represented in the weights.
Modeling the basics of software engineering was easy, since things like writing code that passes some tests is easy to RL. But for anything not easily RL-able, we’ve basically ran out of human generated tokens to distill from.
Without some breakthrough in generalization, I remain unconvinced that we’ll be able to model the finer details of software engineering by looping the existing training corpus back into itself. Garbage in, garbage out.
1 reply →