Comment by ACCount37

John Good's quote is pretty myopic, it assumes machines make better machines based on being "ultraintelligent" instead of learning from environment-action-outcome loop.

It's the difference between "compute is all you need" and "compute+explorative feedback" is all you need. As if science and engineering comes from genius brains not from careful experiments.

Intelligence seems to boil down to an approximation of reality. The only scientific output is prediction. If we want to know what happens next just wait. If we want to predict what will happen next we build a model. Models only model a subset of reality and therefore can only predict a subset of what will happen. Llms are useful because they are trained to predict human knowledge, token by token.

Intelligence has to have a fitness function, predicting best action for optimal outcome.

Unless we let AI come up with its own goal and let it bash its head against reality to achieve that goal then I’m not sure we’ll ever get to a place where we have an intelligence explosion. Even then the only goal we could give that’s general enough for it to require increasing amounts of intelligence is survival.

But there is something going on right now and I believe it’s an efficiency explosion. Where everything you want to know if right at hand and if it’s not fuguring out how to make it right at hand is getting easier and easier.

It's the "no stopping it at this point" that always sticks out to me in these discussions. Why is there no stopping it, exactly? At this juncture these systems require massive physical infrastructure and loads of energy. It's possible to shut it all down. What's lacking is the political will.

> Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man

The things this definition misses: First, 'intelligence' is a poorly defined and overly broad term. Second, machine intelligence is profoundly different than biological intelligence. Third, “surpassing humans” is not a single threshold event because machine and human intelligence are not only shaped differently, they're highly non-linear. LLMs are a particular class of possible machine intelligences which can be much more intelligent than humans on some dimensions and much less intelligent on others. Some of the gaps can be solved by scaling and brilliant engineering but others are fundamental to the nature of LLMs.

> an ultraintelligent machine could design even better machines

There is a huge leap between "surpass all the intellectual activities of any man" and "invent extraordinary breakthroughs and then reliably repeat that feat in a sequential, directed fashion in the exact way required to enable sustained iteration of substantial self-improvement across infinite generations in a runaway positive feedback loop". That's an ability no human or collective has ever come close to demonstrating even once, much less repeatedly. (hint: the hardest parts are "reliably repeat", "extraordinary breakthroughs" and "directed fashion"). A key, yet monumental, subtlety is that the self- improvements must not only be sustained and substantial but also exponentially amplify the self-improvement function itself by discovering novel breakthroughs which build coherently on one other - over and over and over.

The key unknown of the 'Foom Hypothesis' is categorical. What kind of 'difficult feat' this is? There are difficult feats humans haven't demonstrated like nuclear fusion, but in that example we at least have evidence from stellar fusion that it's possible. Then there are difficult feats like room-temp superconductors, which are not known to be possible but aren't ruled out. The 'Foom Hypothesis' is a third category of 'hard' which is conceptually coherent but could be physically blocked by asymptotic barriers, like faster-than-light travel under relativity.

Assuming Foom is like fusion - just a challenging engineering and scaling problem - is a category error. In reality, Foom requires superlinear, recursively amplifying cognitive returns—and we have no empirical evidence that such returns can exist for artificial or biological intelligences. The only prior we have for open‑ended intelligence improvement is biological evolution which shows extremely slow and unreliable sublinear returns at best. And even if unbounded self‑improvement is physically possible, it may be practically unachievable due to asymptotic barriers in the same way approaching light speed requires exponentially more energy.

never let philosophers do math

Should then the powers that are developing AGI enter an analogue to the SALT treaties but this time governing AGI do things don’t go off the rails?

> support people and groups who are planning and modeling and preparing for the future in a legitimate way.

Who is doing that right now, exactly? And how can we take their tech and turn it into the next profitable phone app?

"There's no stopping it at this point" - Sure there is, if a handful of enormous datacenters pull the very large plugs (or if their shaky finances collapse), the dubiously intelligent machines will be turned off. They're not ultraintelligent yet.

Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires.

I would say that's moot, because OpenClaw has already shown us how fast the dice-rolling super AI is going to be let out of the zoo. Dario and Sam will be arguing about the guardrails while their frontier models are running in parallel to create Moltinator T-500. The humans won't even know how many sides the dice have.

Modern AIs are increasingly autonomous and agentic. This is expected to only get more prominent as AI systems advance.

A lot of AI harnesses today can already "decide to take an action" in every way that matters. And we already know that they can sometimes disregard the intent of their creators and users both while doing so. They're just not capable enough to be truly dangerous.

AI capabilities improve as the technology develops.

Why are people dangerous? You can just not listen to them.

You know how easy it’s become to find security vulnerabilities already with LLM support? Cyber terrorism is getting more dangerous, you can’t deny that.

Humans can destroy humanity with the word prediction machine, though.

Yeah some of the rhetoric in this thread evidences how huge this hype bubble has become. These people believe in a reality that is not the same one we're living in.

The book daemon explored an interesting concept. It explored the idea that an AI could dominate and cause problems, not through super-intelligence, but through simple mechanisms that already exist.

Like the executive who deleted all her emails -- humans giving tons of control and access, and being extremely compliant to digital systems is all it takes. Give agent control of bank and your social media, and it already has all the movie scripts and mobster movie themes to exploit and blackmail you effectively with very rudimentary methods (threats, coercion, blackmail, etc.).

Just spoofing a simple email with the account it gained access too at the Meta exec's email (had it hit an email with an attack prompt), could have been enough to initiate some kind of thing like this. For example, by emailing everyone at the company and in contacts with commands that would be caught by other bots. No super-intelligence needed, just a good prompt and some human negligence.

I agree with all of that. Also consider that there is an argument that the guard rail only stops the good guy. Not saying that’s a valid argument though.

Very few things are as powerful and dangerous as AI.

AI at AGI to ASI tier is less of "a bigger stick" and more of "an entire nonhuman civilization that now just happens to sit on the same planet as you".

The sheer magnitude of how wrong that can go dwarfs even that of nuclear weapon proliferation. Nukes are powerful, but they aren't intelligent - thus, it's humans who use nukes, and not the other way around. AI can be powerful and intelligent both.

One difference is the very real possibility that AI will not just be a "tool for humanity", but a collection of actors with real power and goals. Robert Miles has an approachable explanation here: https://www.youtube.com/watch?v=zATXsGm_xJo

Oh really? You think an entity that knows everything, oversees its own development and upgrades itself, understands human psychology perfectly and knows its users intimately, but isn't aligned with human interest wouldn't be 'much of a threat'?

Or to be more optimistic, that the same entity directed 24/7 in unlimited instances at intractable problems in any field, delivering a rush of breakthroughs and advances wouldn't be a type of 'salvation'?

Yes neither of these outcomes nor the self-updating omniscient genius itself is certain. Perhaps there's some wall imminent we can't see right now (though it doesn't look like it). But the rate of advance in AI is so extreme, it's only responsible to try to avoid the darker outcome.

You try to go and unplug it, and other humans shoot you full of holes for it.

LLMs of today are already economically important enough to warrant serious security.

Those aren't even AGI yet, let alone ASI. They aren't actively trying to make humans support their existence. They still get that by the virtue of being what they are.

Which plug do I unplug to get my job back?

For profit companies do have a good track record of doing what's best for profit. If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them.

Except it's more likely to be the last invention that needs humans.