Comment by amaterasu

I'm trivialising, but a lot of software in medical devices is turning a GPIO pin on/off in response to another pin, then announcing that it did so. The piece missing from the article is that the assumed probability of software/firmware (or anything really) failing is 1.0. Everything is engineered around the assumption that things (_especially_ software) WILL fail and minimising the consequences when they do. LLM's writing the code will happen soon, it's a GPIO pin control after all. LLM's proving the code is as safe as possible and that they have thought about the failure modes will be a while.