Comment by usrbinbash
3 years ago
> There seems to be more consensus around 1 now than there was even 3 years ago
Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.
The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.
Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.
> "have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly."
Einstein could be convinced to tell you that 7 + 4 = 5, would you think that rules out him being unusually intelligent? Why in principle wouldn't a superintelligence be able to lie to you? Why in principle wouldn't a superintelligence be able to pretend to fall for a prompt injection attack to keep you from killing it while it improved its position?
> "We cannot even accurately define "intelligence""
Our inability to define intelligence is not something that will stop one existing. Ants can't define nuclear weapons, but nuclear weapons exist. The point of the recursively self-improving scenario is that humans don't have to understand it, or design it, so not being able to define it accurately can't be an objection to how recursive self-improvement is impossible - like saying that uneducated laborers can't get big muscles because they don't understand progressive overtraining and muscular hypertrophy. Their muscles self-improve regardless.
> "how can we know if a superintelligence would be a threat?"
Since we can't accurately predict the future, how can we know that anything in the future could be a threat? Why should we take any precautions against anything? It could be completely pointless, everything might never happen.
> Einstein could be convinced to tell you that 7 + 4 = 5
I think Einstein would have laughed at me if I tried to convince him to do that.
Because other than an LLM, Einstein knew what these symbols denote, what their relation to reality is, and how math works. Einstein didn't mimick math by completing sequences of tokens, and relying on humans antropomorphizing the sequence completion engines output to an actual understanding of the topic.
> Ants can't define nuclear weapons, but nuclear weapons exist.
Ants also cannot build nuclear weapons, nor create anything that would make the emergence of nukes any more likely, among other things because they don't have the ability to define them. So if we accept this premise, then the discussion is moot: We can be fairly certain that we are the most technologically capable entities on this world, so unless we can understand a technological creation to the extend that we can bring it about, nothing else will.
In short: If we are ants to the superintelligence, then we have nothing to worry about, because we likely lack the understanding and ability to create it, or even something that could act as its precursor. If we are not ants, then we should be able to predict when this can happen.
> Since we can't accurately predict the future
We can accurately predict a lot of things. Global warming is an example. And the things that we can predict, and determine how likely they are, we can and should prepare for.
AI doomsday preparation demands the exact opposite: That we prepare for something that we cannot predict, and cannot demonstrate if it is possible, or how likely it is. That's like asking to prepare for an ice age. Theoretically an ice age is possible on this planet, however nothing we can see, measure and demonstrate right now, supports the prediction that an ice age is about to destroy us.
Einstein was not hobbled by having to tell the truth. He was capable of joking, playing a prank, doing it as a favour, doing it as a challenge, as an experiment, exploring the scenario, etc.
> "unless we can understand a technological creation to the extend that we can bring it about, nothing else will."
Where did human intelligence come from? Are you a Creationist? Self-improving AI brings itself about. With the right feedback loops and the right software, the fear is that an AI will grow itself - and no humans and no aliens are needed up front to design it. People are trying to make machines behave like people, like pets, like the world, and emerging out of this is machines which behave more and more like people with every passing year.
> "we should be able to predict when this can happen."
Who says we can't? Ray Kurzweil has been predicting it will happen by around 2030 for years and years.
1 reply →