Comment by _carbyau_

Again, engineers know what needs to be tested, how, to what degree, with which materials, and with what safety margin.

Your widget that mankind may never understand. Well, that's just like anything else in the world. Engineers didn't understand building foundations to anywhere like the degree we do now. Or how to build earthquake proof buildings for a long time. It didn't stop stuff getting built. But it sure affected the confidence in it.

And when they honestly don't know, they don't have to approve it.

I cited the Titan sub example because one engineer refused to sign off on it and was fired for it. Whoever signed off on it has deaths to weigh on their conscience/reputation - for whatever societal good that is worth.

If the latest electric vehicle car never needs to charge but might catastrophically explode randomly between 2-15 years, then the car company doesn't get to take money for the car but handwave away the deaths caused. Legally someone is on the hook. In this case, the engineer who signed off on the magic involved - whether they understood it or not - is at fault.

But honestly, people sign off on shit they don't understand all the time. It's a matter of:

1. did their gamble not pay off?

2. were they negligent?

If your answer to 2. is "the LLM made a mistake I never checked for in regulation approved ways", then you are negligent.