Comment by netdevphoenix
19 hours ago
> Or I fixed a bug in a linux scanner driver. None of these I could have done properly (within an acceptable time frame) without AI. But also none of there I could have done properly without my knowledge and experience, even with AI
There are some things here that folks making statements like yours often omit and it makes me very sus about your (over)confidence. Mostly these statements talk in a business short-term results oriented mode without mentioning any introspective gains (see empirically supported understanding) or long-term gains (do you feel confident now in making further changes _without_ the AI now that you have gained new knowledge?).
1. Are you 100% sure your code changes didn't introduce unexpected bugs?
1a. If they did, would you be able to tell if they where behaviour bugs (ie. no crashing or exceptions thrown) without the AI?
2. Did you understand why the bug was happening without the AI giving you an explanation?
2a. If you didn't, did you empirically test the AI's explanation before applying the code change?
3. Has fixing the bug improved your understanding of the driver behaviour beyond what the AI told you?
3a. Have you independently verified your gained understanding or did you assume that your new views on its behaviour are axiomatically true?
Ultimately, there are 2 things here: one is understanding the code change (why it is needed, why that particular change implementation is better relative to others, what future improvements could be made to that change implementation in the future) and skill (has this experience boosted your OWN ability in this particular area? in other words, could you make further changes WITHOUT using the AI?).
This reminds me of people that get high and believe they have discovered these amazing truths. Because they FEEL it not because they have actual evidence. When asked to write down these amazing truths while high, all you get in the notes are meaningless words. While these assistants are more amenable to get empirically tested, I don't believe most of the AI hypers (including you in that category) are actually approaching this with the rigour that it entails. It is likely why people often think that none of you (people writing software for a living) are experienced in or qualified to understand and apply scientific principles to build software.
Arguably, AI hypers should lead with data not with anecdotal evidence. For all the grandiose claims, the lack of empirical data obtained under controlled conditions on this particular matter is conspicuous by its absence.
It's incredible that within two minutes after posting this comment is already grayed out whereas it makes a number of excellent points.
I've been playing with various AI tools and homebrew setups for a long time now and while I see the occasional advantage it isn't nearly as much of a revolution as I've been led to believe by a number of the ardent AI proponents here.
This is starting to get into 'true believer' territory: you get these two camps 'for and against' whereas the best way forward is to insist on data rather than anecdotes.
AI has served me well, no doubt about that. But it certainly isn't a passe-partout and the number of times it has caused gross waste of time because it insisted on chasing some rabbit simply because it was familiar with the rabbit adds up to a considerable loss in productivity.
The scientific principle is a very powerful tool in such situations and anybody insisting on it should be applauded. It separates fact from fiction and allows us to make impartial and non-emotional evaluations of both theories and technologies.
> (...) you get these two camps 'for and against' whereas the best way forward is to insist on data rather than anecdotes.
I think that's an issue with online discussions. It barely happens to me in the real world, but it's huge on HN.
I'm overall very positive about AI, but I also try to be measured and balanced and learn how to use it properly. Yet here on HN, I always get the feeling people responding to me have decided I am a "true believer" and respond to the true believer persona in their head.
Thanks for pointing these things out. I always try to learn and understand the generated code and changes. Maybe not so deep for the android app (since it's just my own pet project). But especially for every pull request to a project. Everyone should do this out of respect to the maintainers who review the change.
> Are you 100% sure your code changes didn't introduce unexpected bugs?
Who is this ever? But I do code reviews and I usually generate a bunch of tests along with my PRs (if the project has at lease _some_ test infrastructure).
Same applies for the rest of the points. But that's only _my_ way to do these things. I can imagine that others do it a different way and that the points above are more problematic then.
> I always try to learn and understand the generated code and changes
Not to be pedantic but, do you _try_ to understand? Or do you _actually_ understand the changes? This suggests to me that there are instances where you don't understand the generated code on projects others than your own, which is literally my point and that of many others. And even if you did understand it, as I pointed out earlier, that's not enough. It is a low bar imo. I will continue to keep my mind open but yours isn't a case study supporting the use of these assistants but the opposite.
In science, when a new idea is brought forward, it gets grilled to no end. The greater the potential the harder the grilling. Software should be no different if the builders want to lay a claim on the name "engineer". It is sad to see a field who claims to apply scientific principles to the development of software not walking the walk.
>Arguably, AI hypers should lead with data not with anecdotal evidence
This reminds me of people who get sad when they realize they haven’t discovered anything amazing.
I am pedantic and “people that” → “people who” (for people, who is preferred).
Why would you ever, outside flight and medical software, care about being 100% sure that the change did not introduce any bugs?
Because bugs are bad. Fixing one bug but accidentally introducing three more is such a pattern it should have a name.
They are. And we have processes to minimize them - tests, code review, staging/preprod envs - but they are nowhere close to being 100% sure that code is bug free - that's just way too high bar for both AI and purely human workflows outside of few pretty niche fields.
1 reply →
I propose "the whack-a-hydra" pattern
1 reply →
Because why would you make something broken when you could make something not broken?
Because it's way too high bar to be 100% sure outside of few niche fields.
> 1. Are you 100% sure your code changes didn't introduce unexpected bugs?
How often have you written code and been 100% your code didn't introduce ANY bugs?
Seriously, for most of the code out there who cares? If it's in a private or even public repo, it doesn't matter.