Comment by witx

And they are still crap at generating code.

At my previous company they have a series of smoke tests for models mostly focused on performance and architecture. I'm still on the team chat and results just came in:

- It's abohrrent at c++: it keeps generating code with data races and, more rarely, use-after-free bugs! It doesnt seem to be able to reason about lifetimes. This on a mostly mid/junior team. It's a bug fest.

- architecture in c++ is a verbose and layered mess even for simple things, which paired with the previous bugs I mentioned is scary.

- For rust obviously there's no use-after-free, but has same architecture pitfalls of layers upon layers. It uses copy and clone all over and performance is bad. Trying to unwrap all that is messy and costs lots of time. Once in a while it generates unsafe code for some non obvious reason

The scary stuff is non determinism. You get different depending on who prompts the agent but there's always some flavour of the points mentioned above. Funny that my team was very adamant on AI-first (why I left) and now they writting more and more code by hand after some very serious bugs and, as they say, dead moments where they have to wait, sometimes hours, and start wondering about the value of their skills, for the model to generate the next spaghetti recipe