Comment by root_axis

Once we get computers as smart as a mouse, we'll be at most 3-5 years or so computers as smart as a human. We will have solved at the major challenges in AGI and it will simply be a scale problem then.

Saying we don't have mouse level AGI is simply saying human AGI is greater than 5 years away, which isn't a remotely contentious statement.

The difference in intelligence between an amoeba and a mouse is enormous compared to a mouse and us. People greatly under appreciate how intelligent and close to human a mouse/bird/pig are in the grand scheme of things. Emotions, behaviors, motivations, goal setting, memory it's all there already. A flat worm, an ant, a fly - those are the large stepping stone accomplishments.

Think about rate of very long distance communication in humans. It took us tens of thousands of years to get to 2.4 kps dial up modems, and only few decades to get to common 300Mbps. The important signal is seeing a 100bps modem, not a 100Mbps connection.

So the real question is how long until we can replicate a worm's intelligence?

I think you're right. I haven't read the article, but I think Kurzweil is perpetually too optimistic about what we can achieve, at least in how fast it will happen. There are cool things happening, and lots of advances being made, but we are nowhere near anything that could really be called AI.

Plus, I think all the marketing use "AI" is giving a very distorted and inflated view to the average person of what software is actually doing, and what it's capable of.

It's a buzzword, full stop.

What does it mean to be as smart as a mouse? If you specify a handful of tasks that demonstrate it, someone will be able to purpose-build an "AI" to do those things well.

I think if we truly do create AI we will, to some extent, have stumbled into it, but that's OK. I think one of purposes of AI research should be to refine our definition of what intelligence is, because we don't have a very good one.

Perhaps, one day we'll "accidentally" figure out how to make it, and then we try to figure out what it is, because figuring out what it is in wetware hasn't been easy. Or maybe we'll figure out how to make it, but never really understand what it is.

It seems to me, that if we ever come up with AI that can exceed human intelligence (whatever we choose that to mean), that we might not ever be able to understand completely how it really works.

Even more interesting, if we were to achieve this AI, we also might not be able to make use it of it, because if it's truly "intelligent", then it will have a free will, and it might not wish to cooperate with us.

I think you are massively anthropomorphizing.

Every day I run programs that happily die alone on their own. Painlessly. All of the "pain" we feel is an artifact of our evolution, same with having a will to live. The only reason we animals fight so hard to stay alive is that animals which didn't died off, ergo, only animals with a will to live survived and reproduced. Those same forces don't apply to computer programs.

Even the concept of "terror." Who is going to program terror in? What benefit would it have? Why not wire programs to be "happy" when helping us?

It’s exactly how a newborn lives for many months. Of course they have coping mechanisms (adult tenderness, the relief of eating), but life seems to be both incredibly interesting and at times terrifying for them.

There's also the question of mental stability. Once we've given the keys to the kingdom to our new super-intelligent and wonderfully wise AI, nothing prevents it from developping dementia, of forms we may not recognize. It may become psychopatic, delusional, starts having hallucinations.

There is no reason to believe a human-level AI would not develop mental problems just like us. Given their unlimited lifespan, it could be inevitable.

It's entirely speculative but doesn't the model Richard Dawkins explains for the laymen in The Selfish Gene suggest something that would work - a drive to reproduce at the lowest level resulting in some incredible things happening over a long enough period of time.

The way I understood the comment, even in that scenario of the machine self-terminating, the point grandparent was making still stands.

Nothing in two million years of evolution programmed us to do programming. There is no common abstraction between it, and ape-on-the-savanna type problems.

Yet, nearly every human being can be taught how to program... But we aren't anywhere close to building an AI that can.

also, why assume computer intelligence is going to work similarly to human intelligence? I would assume that one potential way for it to work is to run simulations using data from sensors in the environment, and make more accurate predictions on an effect an action would have.

If such a machine can exist (and i believe it can, as computing power increases), then AI surely will follow sooner rather than later.

Perhaps, but that's a big if. We can't even say for sure whether a complete reverse-engineering of neurons needs to take quantum effects into account, as Penrose has suggested.

All you're saying is that if we knew exactly how humans work, we could build one. Seems like a tautology to me.

If I knew the exact quantum state of the Universe at the Big Bang, I could figure out exactly how the Universe evolved, but that's never going to happen either.

I think the complete reverse engineering the way you are describing will not be possible. We can only try to reproduce the same outputs for the same inputs. But I don't think we'll be able to fully define what happens in the black box in between.

We might come up with something that works similarly, and can do great things with it, but I don't think AI can be invented the way you are describing.

What you describe is a very high-dimensional non-linear system. We don't have the mathematical tools 'know' such systems. We 'know' a system when we can describe it by a much simpler (preferably mathematical) model. This is why linear systems are easy - we have mathematical tools to break it down to simpler parts (reductionism) and then understand the whole.

If the best we can do with the brain is simulate it at the the level of connectome/neurons/synapses/ thereby creating a system as complex as the brain - then do we really 'know' it ?

What is difference between Conscious AI and General AI?

Even Penrose will tell you he's not certain on this being "fundamental". It's a theory he is working with.

Computer people really hate Searle and Penrose, because they spoil the beautiful picture of Strong AI.

The manner of human "pre-programming" is so abstract that it falls outside the definition of what is normally meant by "pre-programming" in the context of AI. If you had some basic "instincts" codified into your AI that allowed for "self-perseveration" or was manifested as some type of reasoning-engine-BIOS that's one thing, but applying arbitrary training data sets onto your AI (as a pre-requisite for it to function maximally) is the type of "pre-programming" I'm getting at.

Primates have relatively few instinctive behaviours. Almost everything we do is learned to one degree or another.