Comment by Smaug123
3 hours ago
"Yo human, you asked me to do X; I can do X, but I strongly suspect you don't want me to, because it's illegal and it has these consequences. Confirm you want me to do X?" would have been a start, in this case.
3 hours ago
"Yo human, you asked me to do X; I can do X, but I strongly suspect you don't want me to, because it's illegal and it has these consequences. Confirm you want me to do X?" would have been a start, in this case.
With humans (and some machines) we tend to put/manadate additional processes for certain risks instead of just relying on their own good nature. Why just rely on the machine when elsewhere we have learned not to necessarily just trust them so much?
Because we should not settle for building a world in which every interaction must be assumed adversarial! Obviously risk reduction processes are good because they reduce risk, but we should not accept building entities which are actively trying to defeat us (which is what happens by default).
I don't want such a world either.
So far I'd say these entities are hypothetical (unless you include a lot of other machinery that does unexpected things at times - but then it's a different discussion).
I generally think we are quite good a policing really dangerous things (I think the bioweapon convention is a good example of people agreeing that certain risks not being worth taking).