Comment by gcoakes
13 hours ago
I drastically improved the reliability of an internal IoT device so it can tolerate all kinds of abuse (abrupt power loss during update, unreliable SD card, bootstrapping over network when the SD card is missing/damaged, fail-safe updates, recovery in a network isolated environment). This is the first release that I really leaned into AI, having been pretty skeptical before.
I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
> I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
It always works best when layering on top of good human code. The thing is, you start to run out of that as you lean in. Tricky.
Heh, you should look at the reply I left for your sibling comment. I think it is just a regression to the mean of all open source software upon which the models were trained. Sometimes the mean is an improvement.
Hell, if the Dunning–Kruger effect is to be believed, it would be bold me to assume I am capable of good human code. Maybe the average is the best I'll ever get.
> The thing is, you start to run out of that as you lean in. Tricky.
I think it depends how you lean in. I generated a few thousand lines of Rust code today. It was actually pretty high quality. On my initial review, I only identified one change I wanted to make.
Most functions were pretty short and well factored. I wonder if using Rust as a target helps, because (1) the training data doesn't include all the terrible Javascript and Python etc. found on the internet, and (2) Rust and many of its libraries encourage good architecture - code factored into traits, functional pipelines, a powerful type system, etc.
LLMs trained on bad code for mediocre languages naturally write corresponding quality code.
A big part of it still boils down to prompting - what you're asking it to do and how much freedom you give them. If you let them make all the architectural decisions, you're likely to end up with issues.
> It can be used to make existing products better and more secure. You just won't naturally hear about that.
Why? I keep asking for examples of LLM being used to make code more simple and elegant (which I would say is adjacent to making software better, if not more secure), and never get a reply, not a single one. I'm not claiming it doesn't exist, I know it must exist. If only via a lots of careful massaging of system prompts and harnesses, even if it takes too much effort to be worth it, I would like to see it.
Of course people can't share internal examples, but this is all over the world for many months now, where are the open source examples? Not "something new someone couldn't have done before", but someone making their precious even neater (while also adding things to it, but not primarily). Put differently, can LLM delete code, can they curate it well, can they not write weird comments nobody wants or needs, that sound like echoes of the user prompts and responses to them, but just explain tricky parts of the code, like a human would?
Also, IMO a VERY useful thing to learn about a tool or a chemical or whatever is, "what happens if I keep applying it to the same small thing over and over?" -- because that's the level of scientist I never evolved past, the kind of stuff you do as a kid -- , whereas a lot of this seems to be more about how it feels to apply it here and there until it looks good, and then move on and never look back. Which really isn't much at all, especially given we're such a short time in. I mean, how useful were mugs made of lead? Are people aware of the (in hindsight) insane things that were done with x-rays as those were invented, before the potential for harm was known? I bet they had a great time playing with that stuff, for a while.
What if in 5 years it turns out that a few years ago, LLM got into this habit of implementing certain things in a certain way because it seemed the best to "them" (the training, whatever), but there's a security issue with that -- what do you do? You can only let more agents loose, and hope they don't miss anything, because you cannot even do something like "patch PHP" (which is not my thought, Casey Muratori mentioned that in on of those podcast things).
Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though. It just does, like suggesting we go to Mars, because everything is better there, and to do that, we have to blow up Earth. Or saying to magically be able to fly from now on, we need to saw off the branch we're sitting on. Even if I thought life on Mars was the future or that we could fly, if we just thought of it the right way, this isn't a way forward IMO.
> LLM being used to make code more simple and elegant
Simple and elegant generally doesn't come from the LLM. It comes from me saying no to the LLM. I've discarded many branches that didn't feel like they were going the right way. Hell, I aggressively do that with my own code also. I rarely commit the first thing that works.
> where are the open source examples?
My general impression of open source is that it is, in part, a market driven by good software engineering. There's certainly below average code, but that tends to get mocked out of existence before it grows too much. Proprietary code that doesn't have a strong forcing function used to be able to hide poor engineering principles.
I think AI generated code is a regression to the mean, and the mean can be an improvement sometimes. I inherited this project. It was not originally written with AI. A lot of hard work and good engineering went into the hardware/firmware. The IoT software connecting it to our intranet, less so. Some examples that I'm fixing/fixed: ignoring HTTPS certificates, RCE by design, in-place file updating JSON files, a hand-rolled (but ineffective) fair mutex, busy waiting for a device to appear, and worst of all, no tests. None of these are hard to avoid, and even the small AI models would have come up with a better implementation.
This doesn't seem unique to my project either. I think a lot of software is just silently getting closer to the average of all open source code upon which the models were trained. I don't think I want things like SQLite to get closer to the average.
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I think you might be reading too much into what I said. When I say "fully lean into", it's probably less than you're thinking of. I still feel that I understand what was written and can stake my reputation/career on it.
You want them to make code more simple and elegant by themselves, no other direction? "Make this code simple and elegant, no mistakes, no hallucinations"? I don't think that's realistic at the moment.
But with suitable direction, they can definitely improve codebases - especially a typical enterprise codebase, which, if you don't like LLM code, you would hate with a passion. (I can only assume you've never seen such code otherwise your perspective would be different.)
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I essentially agree, but there's a huge spectrum before you get to that point, and lots of ways to use it effectively that doesn't create tech debt.
> You want them to make code more simple and elegant by themselves, no other direction?
No, he wants a project where AI is used as a major help, but the result tend towards good code, i.e readable and well abstracted. Even if the cost is high, just an example where you can extract the best practices of using AI coding.
2 replies →