Comment by miyoji
11 hours ago
> If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me
This is a very bad analogy because chess isn't life. In chess, you aren't allowed to do whatever you want. There are rules. I know for a fact that Magnus Carlsen won't beat me using checkers moves and he won't beat me by pulling out a gun and telling me to resign. Magnus Carlsen's skill at chess leading to his victory in chess is not a valid analogy here, because there's no law of nature that says "the more intelligent entity wins in a battle for survival".
You could have infinite superintelligence and still die inside a locked room to which you have no key. "Superintelligence" is not a magic solution to every problem, you can constrain any superintelligence with any unsolvable problem.
I’m not sure how you came to believe there aren’t rules in life, but there absolutely are. As you point out, there are physical constraints on everything and if you find yourself in a concrete box with nothing but a DGX H100 or at the bottom of the wrong gravity well, there’s nothing you can do. Checkmate.
This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?
> I’m not sure how you came to believe there aren’t rules in life, but there absolutely are.
This is why I fucking hate commenting on this website. You know exactly what I meant but you wrote this bullshit anyway.
I knew you weren’t dumb enough to think that the physical laws of the universe don’t exist, so yes, that sentence was a bit of rhetorial flourish. But I genuinely don’t understand what point it is you thought you were making and your entire argument seemed a bit muddled.