Comment by bbor
3 years ago
I’d like to push back, if you have the time to clarify. Are the following summaries accurate renditions of your points?
1. We shouldn’t worry about SI because we haven’t seen one yet. 2. We shouldn’t listen to smart people, because they’ve been wrong before.
Because if so I obviously don’t find those convincing. The whole reason the SI cultists are so “alarmist” is because by definition this is the kind of problem we have to preempt, not run into and then wing it.
If someone responded to concerns about the atmosphere igniting into nuclear fire with “that’s never happened before and the only people worried about it are scientists, so don’t worry about it” instead of equations… well I’d be damn well terrified
How do you respond to concerns about fairies stealing your children?
For that matter, how do you respond to a calculation (there was one!) that railroad trains could not exceed 41 miles per hour, or all the air would be forced out of the cars and all the passengers would die?
I believe that jtsiren's point is that we're not at the point where we can even define intelligence. We can't calculate anything, because we can't sensibly define any of the variables in the equations. All we can do is make guesses about terms we can't even define. We're like stone-age people worrying that if their neighbors' cooking fire gets too hot, it is going to light the air on fire and we're all going to die, and nobody can reassure us because nobody even knows what burning really is, or what the air's made of. We're millennia away from being able to do the kind of calculation that you want.
So the only choices available are to proceed, or not. And if your answer is "not", you need to convince everybody, because I don't think we're going to (for example) nuke North Korea to stop their AI program unless you've got a really convincing case. Which nobody has now.
[flagged]