Comment by bigyabai
8 hours ago
So, which do we choose?
A) Let AI override safety mechanisms (increases uptime while endangering humans)
B) Let AI impose additional safeties (decreases uptime with questionable safety uplift)
If you intend to go with option B) then there's basically zero benefit to investing in that kind of automation. AGI will be fallible, and even with all the safeties and safeguards it's hard to see what tangible benefit AGI would add to the system. It'll be a hard sell for critical infrastructure of any kind.
I understand that it's hard to get a man to understand something when his AGI Society executive credentials depend on it, but it feels glaringly obvious why AI isn't masterminding these sorts of systems. Even accounting for unprecedented advancement in AI, it's hard to see what AI could contribute to managing these things.
> It'll be a hard sell for critical infrastructure of any kind
Remember, critical infrastructure attached SCADA systems to the internet and has had trouble removing them all despite 30 years of warnings and audits.
There's definite advantages to having a LLM in the loop. The best we can hope for is that it's architected in a way that it fails safe and damages are limited, so we get some of the advantages but less of the downside.
That's a configuration issue, not people preferring a bad idea. SCADA itself is not a hard sell because it's ostensibly deterministic.
If AI cannot provide the same baseline promises while only bring right a fraction of the time, then it's not the right solution. I don't want my car's AC to ask Anthropic what the right temperature is; it's a waste of tokens that isn't optimizing jack shit 99.99% of the time. If a system isn't failing currently, then it's probably in the same boat as car ACs. Shoehorning AI into it won't solve anything.
There are more than two options
Okay, explain your third path.
I suggest you research organizations like IAEA and EAA which create communities around managing the problem of “scary new technology” of nuclear and aviaton respectively (two just for example).
There are ways to manage “scary new technology” that are well trodden,
It does require people to let go of whatever pitard they have decided to commit to.