Comment by Ericson2314
6 hours ago
No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief!
Next up is demonstrating the AI is more productive with better programming languages.
I've already seen at least one promising experiment about how static checking helps LLMs: https://arxiv.org/abs/2606.01522
Key parts of the abstract:
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
> Next up is demonstrating the AI is more productive with better programming languages.
That actually would be pretty exciting!
But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material?
Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.
That’s not necessarily a bad thing.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
> That’s not necessarily a bad thing.
True, and to my point, "popular" isn't necessarily "better".
> because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances
Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
For sure. Its code output is definitely not to be fully trusted. Oftentimes, you can point at where it made a mistake and it usually fixes it too.
i can't wait for that one haha