Comment by nearbuy
8 hours ago
> The problem comes from stipulating that something with a negligible probability has a high probability.
We are not doing so, and I don't know how I could have been more clear that we are not saying this hypothetical will happen. Would it help if the hypothetical was that the AI knows a magic spell that blows up the Earth?
It's a simple question. Would you think AI censorship is acceptable if the information actually were dangerous? Don't tell me why the hypothetical is impossible because that's entirely missing the point. I don't know what your position is, and so I don't know what you're arguing for. I don't know if you consider freedom of information to be a terminal virtue, or if you think it's good only when the consequences are good. Telling me the hypothetical won't happen doesn't clarify anything; I already know that.
You can have the view that we only want freedom of information when it causes net good, and that it always causes net good. Or maybe you have the view that freedom of information is always virtuous and we shouldn't consider the consequences. Or maybe something else. Until you clarify your view, I don't know if/what we disagree about.
Hypotheticals like that are uninteresting because there are only two ways it can go. The first is that you can find a way out of it, and then you say, do we need the magic spell for anything? Is knowing about it useful to preventing it from being used? Then people need to know.
The second is that you're stipulating the information being available is going to destroy the world with high probability and no possible means of mitigating it. Then anything else gets drowned out by the end of the world, but only because you're stipulating the outcome.
Which you can't do in real life, not just because the real probability of the hypothetical is so low but because there isn't anyone who can be trusted not to fudge the numbers when they want to censor something. Should it be censored if there is an absolute certainty it will destroy the world? There isn't much room to move in that one. Should it be censored because somebody claims it's really bad? Nope, because it's way more likely that they're full of crap than that it's actually going to destroy the world.