Comment by fragmede

16 days ago

why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.

Why do you assume that AI is gonna nerf everything?

13 comments

fragmede

Reply
AnimalMuppet  16 days ago

AGI might. AI? No way.

See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?

That's not going to nerf anything.

  • deathanatos  16 days ago

    > See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?

    The generalization of this is why I think all these AI companies writing blog posts where the marketing department is just jer—ranting endlessly about how AI will improve itself into the singularity is just crazy talk. They generate a random statistically likely output, and the most statistically likely output is mid. Exceptional outputs — the ones that wow us or move the needle are exactly that, unlikely. AGI is sci-fi, and LLMs will not change that.

    You can see the same effect when AI emits bash, too, and especially so since most bash is terrible, and most users of bash do not put in the effort to learn bash and its foibles. So it outputs what most people write, which is not great.

  • CamperBob2  16 days ago

    When's the last time you saw a decent coding model create a buffer-overflow bug while trying to use C strings?

    Serious question. Anyone else seen this happen in the last 12-18 months? If so, which model and version were you using?

    • smackeyacky  16 days ago

      I had Claude write a bit of stupid C# the other day that had an off by one string truncate. Surprised the hell out of me.

    • macintux  16 days ago

      Would you even know? Serious question. The volume of code the models can produce, the subtle ways these bugs can manifest (or even only manifest when under attack), it seems like they would be easy to overlook.

      3 replies →

    • smj-edison  16 days ago

      I use Zig, which has slices, so so far none. But man, it can't get ref counting right to save its life. There have been remarkably few times it's gotten it right on the first try. My codebase considers OOM recoverable, so it keeps forgetting to clean up memory when OOM is raised. Even in the happy path though it still messes up ref counting. I use Kimi k2.6.