Comment by NumberWangMan
3 years ago
> The Argument From Stephen Hawking's Cat
> Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it?
> He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.
A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him.
AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.
That's true and relevant, but doesn't prove much. What's the big plan, what's the next step in the plan?
Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted".
It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.
Can we detect companies doing bad things, today? Yes, and we still have a hard time containing them and their externalities.
AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help?
That's the big question IMHO.
>Yes, and we still have a hard times containing them
Only because we're most of the time not properly motivated because there's no urgency. When a real threat, like a war or a virus say, comes around suddenly we can organize things pretty effectively.
It resembles the Bill Clinton quote about controlling the internet being like nailing jell-O to a wall. Yet autocratic governments have done that, and not just that but turned it into quite scary systems of control, with little difficulty. Just like 90s cypherpunk fantasies about digital anarchy these AI scenarios are nerd revenge fantasies. The intelligent guy outsmarting the big guy, but in reality physical power always wins.
6 replies →
I find this to be a much more plausible area of concern than standard horror-story AI doomerism. But the more we shift our focus from fantasy to reality, the more we have to acknowledge reality, as Maciej does in this talk. It isn't plausible to prevent all the great powers from developing AI. We don't have that level of cooperation in the world today. We are going to need to adapt to this by keeping up with our adversaries, whether they be hostile nations, corporations, or mad scientists.
4 replies →
>Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints.
Why is it difficult? We are already putting AIs in every electronic device. Someone has probably already put an LLM in a robot somewhere. And you don't think Boston Dynamics is thinking about putting an LLM in one of their robots to test? And surely the military is building AI fighting robots.
And then there's the thought that a super intelligent AI can easily hack its way into any machine it wants.
Heck, the super intelligent AI doesn't even have to convince humans. In the future, it can just convince dumber AIs that are already in robots.
This is like saying "sure, that site is susceptible to a sql injection, but that doesn't mean the whole thing is insecure."
If you have an intelligent adversary and the stakes of them succeeding are high, it is the defenders job to prove the system secure. Systems don't start off secure and become vulnerable - they start off vulnerable until proven secure.
So yes, its okay to say "the things we're doing to 'contain' ais are almost certainly inadequate" until shown otherwise.
This analogy assumes that there exists some relatively straightforward, secure, practical defense against superintelligence that I am arguing against, like protecting a website against SQL injection. I do not argue against such a defense if it exists - by all means research and present proposals. My comment is about the plausibility of AI domination. I don't think it is that plausible, so I have different views than others on how important it is that we restrict the development of AI.
100% agree.
However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.
Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed.
They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find.
Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems.
Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI.
People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case.
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
They’re hard to find because you’re comparing arguments for god to arguments against god.
They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction.
It’s a logic argument.
1. Super Intelligent AGI is possible.
2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.
3. We have no idea how to align AGI.
There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.
People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
35 replies →
Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.
Personally I find people with the superintelligence fetish frustratingly dumb too.
Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
And that's before we even think about whether it would even have the motivation.
I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.
> Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.
When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.
What is your heuristic to determine what is likely?
The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering.
Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave.
That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.
You're mixing metaphors in your "trivial rebuttal", to stay with the analogy Hawking would have to hire a cat.
We're allowed to assume the existence of cat carriers in this metaphor, but not the existence of the humans that are the sole reason cat carriers exist?
I was trying to be generous! But you are of course right, in the better analogy Hawking would find himself born into a feline-centric world and face an even harder task.
4 replies →
> AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter.
One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining.
It’s an entertaining conspiracy theory.
And it's not hard for Hawking's hired aides and nurses (who of course existed) to do so, either. As an able-bodied person whose cat very much does not want to go into the cat carrier, it's not that hard, if you use your brains. You feed them a little gabapentin, whose existence they cannot even comprehend (and you know they don't know because you have used your mind to model the cat's behavior like 'do cats understand drugs' or 'do cats like eating treats'), and when they are drugged, you put them in the cat carrier. Done.
Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.
You're positing the existence of a whole society around Hawking, up to and including a pharmaceutical supply chain, where the correct way to think about it would be Hawking waking up alone on a cat planet. I have no doubt that a complex society of embodied hyperintelligent able-bodied beings could outfox humanity, but that's not what we're talking about with this AI risk scenario.
I see your point, but relative capability levels aren't the only relevant factor here, absolute capabilities matter as well.
It seems plausible to me that even if we are to the AI as cats are to us, we've reached an absolute threshold of generality that allows the AI to be confident in our ability to follow simple (to it) instructions, in a way that cats can't for us.
> but that's not what we're talking about with this AI risk scenario.
Yes, it is.
2 replies →
> If you're not allowed to have a bank account, simply barter.
Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.
Current bureaucracies already are "AIs" of sorts, notably "expert systems" (the rule book) with a bit of "temperature tuned" hallucinating at the edges. (The human bureaucrats applying judgement.)
It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.
> with a bit of "temperature tuned" hallucinating at the edges
Hahahaha, that could be too. So we're "legs, hands, eyes, and hallucinating agents" serving our master AI.
I like an image I heard today: life emerged in the primordial sea, and AI emerged in the internet, which is the information sea.
or you know, just pick up the cat and put it in the box.
I've yet to see what would be global news: "it's impossible to put this cat into a carrier"
You're missing an important bit of knowledge about Stephen Hawking.
That he's dead!
True but also, what the hell is going to do with the cat carrier? Velcro it to his wheelchair?