Comment by ForHackernews
1 day ago
AI performance has always been extremely spikey. It's great at some things and terrible at others.
Why do you think the world to date hasn't been taken over by evil genius mathematicians? Can you extrapolate from your understanding of the answer to that question?
I don't think evil mathematicians are very common or that any of them would be capable of single handedly taking over the world if they were. My concern is more about systems that are beyond human level, those kinds of systems would actually be dangerous to us.
I see the recent progress in mathematics and cybersecurity as signs that models are getting more capable more quickly than usual. The companies plans to develop them by recursive self improvement now seems like a real possibility and I don't think they should be allowed to attempt this.
Solving a bunch of math proofs is a far way from recursive self improvement. Don't worry, it's not like a tech tree in a video game where if you can prove a bunch of theorems then suddenly you unlock the next level of technology.
Machines are already far beyond human capability in plenty of ways. Including cognitive tasks like chess. We've already created the technology we need to destroy ourselves (nuclear weapons), and yet so far (knock on wood), we're still around.
We've even already had programs that can prove (brute force) theorems. As far as I can tell this isn't much different, except the space of theorems that computers can solve has expanded. How far? We can't really say yet.
Does solving more theorems than before suddenly mean computers are capable of anything? No.
Lets turn this around, are humans capable of anything? We like to think we are, but that just seems more like our ego than any hard truth.
Do you think that a tireless, infinitely smart, infinitely evil human would be able to take over the world? I don't. Intelligence is not the limiting reagent in our reality.
I think so, individual dictators have gotten pretty far and they weren't infinitely smart. I think infinitely smart would be enough to extend that to the whole world.
> Why do you think the world to date hasn't been taken over by evil genius mathematicians?
A "mathematician" is a human who decided to spend their lives studying mathematics. Mathematicians also tend to be smart, but intelligence is innate, not acquired, so studying mathematics doesn't make you smarter. This makes it obvious why they don't rule the world - if you want to rule the world you'd want to focus on that (for example, doing business or finance), and becoming a mathematician is just a waste of time.
LLMs don't work like that. Like in humans, all of their capabilities correlate, and unlike a human, their overall capabilities grow over time. Looking at LLM mathematical ability over time* therefore gives you info about the progress of their general capabilities, and ability to take over the world would be determined by the latter.
* In fact it'd be better to look at a mix of different capabilities, but that's growing too at about the same rate, see https://epoch.ai/eci
> unlike a human, their overall capabilities grow over time
This is incorrect. Unless there is some new developments I'm unaware of (entirely possible) LLMs "learn" during the training phase, but after that they are static. They do not improve further or retain information when used for inference.
You might be confused because AI companies keep releasing new models and tinkering with the harnesses, sometimes under the same name such that "Zern 6" (or whatever) doesn't always mean the same thing.
Nah, I just phrased it a bit confusingly - I meant the capabilities of LLMs as a technology (equivalently, the capabilities of whatever the frontier model is at each time) grow over time, even though any particular model is static.
They grow over time if you consider a lineage of models as the same model.