Comment by OptionOfT
3 days ago
Well, I'm glad it's that slow. I can't shake the idea of the horrors it would be to get a glucose pump whose software has been vibe-coded.
3 days ago
Well, I'm glad it's that slow. I can't shake the idea of the horrors it would be to get a glucose pump whose software has been vibe-coded.
I work on team managing safety critical code. Management has asked to increase our AI usage, especially for generating requirements.
That’s not necessarily a bad thing. I often encounter badly written or conflicting requirements. An AI may be better at detecting problems or gaps than humans.
Yes and... no.
I find the risk here that the requirements are the average of all requirements, so the exceptional things don't really get highlighted.
Because you now get this giant amount of text shoved in your face, you switch from thinking to validating. Is what's there correct, vs starting from a blank canvas. The doc already curtails your thoughts.
Kinda like all cars are starting to look the same. No one takes risks anymore.
No-one wants to / feels empowered to / has the knowledge to ask the really difficult questions.
I certainly get it. But I also am very frustrated with the snails-pace development of closed loop glucose pump system. The tech has existed for quite some time to implement them in theory. Body hackers have already done so a decade ago.
I often wonder if we have created the correct balance here. How many quality of life years have been lost due to the decades lost by being conservative? And how much of the conservative pace is done for the “right” reasons vs personal or corporate CYA?
It's a question of incentives.
For safety regulators, the incentives are all on the side of limiting acute downside (e.g. a plane crashing), not maximizing potential aggregate upside (e.g. millions of tons of fuel saved per year and millions of tons of C02 not in the atmosphere).
Society punishes regulators that approve products that kill people, so regulators adapt to this and as a result tend to be very conservative.
Regulators don't capture any of the upside (reputational or otherwise) when a new product enters the market and cures disease, makes cars more efficient, helps planes land on their own in an emergency, etc.
I don't know what "right" should be here, but you've hit on a good point. It's complicated.