Comment by TheAmazingRace
5 hours ago
I wonder if we have an AI LLM equivalent to Moore's Law. Like how often do we expect improvement in this technology and with what timing?
5 hours ago
I wonder if we have an AI LLM equivalent to Moore's Law. Like how often do we expect improvement in this technology and with what timing?
Yes -> every 18 months they've gotten 90% more efficient for the same level of quality for about 5 years. There's little sign that trend is slowing. If anything, there's reason to believe that System 1 models (plus potentially 1-2-3 workflows) may increase that over the next 3-5 years.
You'll know when the trend stops -> when the intelligence differential between smaller models like 7B starts to grow instead of shrink from 32B models -> that means 7B is getting about as smart as it can get. Then, 32B will follow next, then 70B, etc etc.
We haven't yet seen that at any size AFAIK.
Andrej Karpathy said once that he expects superintelligence could fit in 1 billion parameters.
Super intelligence that doesn't have to deal with the real world, maybe.
I wouldn't be surprised if less than 1B param equivalent of our brain deals with solving math and writing computer programs and physics and all the things we tend to associate with "intelligence" - especially if you ultra optimized for that, I doubt our brain works like that.
Dealing with the real world, I highly highly doubt it.
2 replies →
According to Epoch AI:
> The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. [0]
0. https://epoch.ai/publications/the-plunging-price-of-thought
Then why are AI plans still so super expensive, and AI spending going through the roof, while all the subsidies are ending?
https://en.wikipedia.org/wiki/Jevons_paradox
AI gets cheaper, people use it everywhere. Google searches, for example. Now we want to crack math problems and spend weeks with unreleased models.
If you used GPT-2, it'd be incredibly cheap. You basically can't use it for anything and it's simple to serve.
The cost per fixed level of intelligence is dropping, but we're also getting dramatically more intelligent models.
Reddit is full of people complaining how they burn their 200$ sub in half an hour by starting ten Max sub agents. That’s to say, many people just don’t know what they’re doing.
Because models are only getting better at a rate of 10% per year, people always want the best quality possible. You can get SotA performance from a year ago for a fraction of the cost, but why would you use Opus 4.5 when you can use Opus 5.5?
Because it's increasingly useful and the thing you are substituting (human time) is much more expensive.
Apart from what others said about using more intelligent models instead of cheaper ones, token usage is also increasing a lot. Classic Jevons paradox
At least for me the Claude plans seem like an incredible deal and I never hit my limit.
reminds me of this blog post: https://campedersen.com/singularity
Whew, at least I won't have to hand-code solutions to the 2K38 problem!
I've heard tell about 100% of certain types of work being ended in batches of six months. For years. Truthfully, I'm skeptical, but accuracy wasn't prioritized.
double the information density every 2 days?
serious bit: if you think about how these smaller models work, at the end of the day it seems that they are now capable of forgetting useless information because they're able to derive it in reasoning allowing models to become smaller at the cost of requiring more reasoning tokens to solve a task.
Knowledge will be shifted to systems like n-gram augmentation which are relatively cheap and will not compete with reasoning capabilities for weight saturation.
Hopefully enough runway for an existing model to train the next to be better than itself with absolutely no human intervention.