Comment by fooker
17 hours ago
Prediction - we are going to figure out SOTA AI performance without requiring 1TB of memory within a year or so.
Of course CXMT, Micron, and family will still be profitable, but maybe not 'surge 470% from IPO' profitable.
How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.
Once I start a conversation about Postgres query plans, maybe 95% of the knowledge will be almost certainly not needed, and for an inference provider there will be many more concurrent queries with reasonably large overlap. Maybe future architectures will be able to take advantage of this so not all parameters are needed in memory for almost all instances.
If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.
Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon.
Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
2 replies →
It doesn't matter. However much intelligence you can squeeze into 1 GB, people will always want more.
Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.
Your example contradicts you.
2K, 4K, and 8K+ TVs have been around forever, and although 4K has become the norm, it's widely accepted that there isn't much benefit for most people above 1080p
8 replies →
Your example also counters your point: A lot of people do not care about 4k, and even less do care about 8k or above. We reached a point where most people are completely happy with the quality they get.
3 replies →
Don't forget that it's only an assumption that scaling more results in better models. There may be a ceiling to that. If that is hit and the best performing possible model can run in little VRAM, your argument here doesn't hold anymore.
In a way it is actually the same thing that makes us accept AI as working in the first place. It only needs to be good enough for human perception. The same is probably true for compute.
3 replies →
FWIW I still play PC games on 1080p (also has the nice side effect that I don't need to buy an overpriced highend GPU, nor use upscaling hacks like DLSS which trash image quality).
> Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.
And yet when we hit 4k, that's were people just stopped buying higher res. 8K is still useful, but only when the screen is so large that it doesn't fit in the room :-/
1 reply →
All the terms are squishy, but being sloppy about it, I think there's intelligence that needs to be in the model, and knowledge that could live in a database. Right now, models are memorizing a lot of stuff they don't need to. We know how to index lots of information on (comparatively slow) SSDs or across a network.
See Karpathy's "Cognitive Core" idea. (I don't have a good link)
I was recently skimming through some 20-30 old books about Information Retrieval (https://en.wikipedia.org/wiki/Information_retrieval), and it was immediately quite striking to me that LLMs are the culmination of the integration and development of many ideas in IR that were impractical or theoretical back then. Decoupling the database from the search interface would make the relationship between modern AI and IR even clearer.
So far, frontier capability keeps getting shrunk to fit consumer level hardware. The lag time being over a year, though, precludes this from being called SOTA by the time it arrives.
There must be a limit, I agree, but there have been no signs of approaching it yet. The most recent cohort of small models have shown the biggest leap in capability so far.
I haven't tried it, but colibri is meant to allow running GLM 5.2 the 744B model in 32GB by using prefetches and streaming from fast SSDs.
https://github.com/JustVugg/colibri
Terrible performance on $20,000 worth of GPUs.
Historical precedent: Quickselect, or any other discovery of a surprising algorithmic speed-up.
LLMs haven’t been on the scene for very long. There’s still lots of room for efficiency discoveries. Plus, I keep hearing quantum is going to be a big deal in the next few years.
Super interesting, when Quantum computing would enable really large models, or much larger contexts. But QRAM is even more behind than pure fault-tolerant gates.
I'm claiming that the max amount of intelligence you get out of a 100GB model is unlikely to be that much lower than what you can get out of a 1TB model.
That depends on whether you can separate intelligence from memorization. I think we can agree that theres more than 1 TB of memorizable information in the world, even when highly compressed.
1 reply →
It’s not just about small models, that’s only one part of evolution
Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size
> Some groups are baking models into silicone
While some other groups are baking silicone into models :)
1 reply →
I think this is the future - at least it will be for on-device models. Apple, for instance, will "bake silicon" once a year for their current model, and use that chip in all their devices.
> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.
Sure, it might be very near using the current approach, but... it might also be might be very far off because we are using the wrong approach.
I mean, look at the max amount of intelligence you can get out of a human brain powered by two bananas...
Though even if there are big further breakthroughs to be made here, they won’t necessarily be made soon, and if they aren’t made in the next ~10 years then they won’t really affect the outcome of chip investments made now or in the immediate future.
> How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing
Maybe? Human science history is absolutely littered with examples of things that were “constrained” by a fundamental law … until they weren’t, because we’d misunderstood or misapplied the law.
Just replicate dogs brain. It is the ultimate state of art (far smarter than people). But you need a few kb of memory to replicate resoning skills.
Dogs are very smart!
And by the time the models that require 1TB memory will be better.
It's like people saying mobile chips are going to be better than PC chips... until they realize PC can be made of mobile chips too if that comes true.
No, it's like saying a chip that consumes 500W is going to be better than a chip that consumes 200W.
This used to be true for the first several decades of microprocessors, and stopped being true in the early 2000s.
The fundamental issue is that for these systems bigger is essentially always better (if affordable). So if we can squash something like Kimi K3 down to run on a 'normal' system, that just incentivizes devs to increase the model size until once again we're at the limit of what can be run.
What do you base this prediction on? It seems very unlikely, unless by SOTA you mean the current SOTA.
> What do you base this prediction on?
Their posterior. Or worse, their interest in going to the moon (HODL style, not Artemis style).
I think the chipmakers are pretty securely set up here, the usual cyclical issues aside. Between the stubborn resistance of the data to being compressed without significant loss, and to a lesser extent the ability to do more with more memory (we’re surely at diminishing returns for LLMs already, but diminishing doesn’t mean zero) it seems like a fairly safe guess (I am not an expert on anything) that for the next decade at least you’ll still want at least 128-256GiB of video RAM to have a comfortably “SOTA-enough” experience. Then the other side of the rectangle comes from putting that VRAM in every consumer and office-worker device. Selling everything at a big premium to a few efficiency-obsessed SaaS providers is what you do when you’re temporarily supply-constrained. Sidelining the SaaS middlemen and selling to gloriously inefficient consumers who will leave their laptops closed for most of the day is the dream. This is probably not unrelated to the fact that the country which is currently making a big strategic push into memory manufacturing is also championing open models.
It sounds like you have strong faith that if we can do in 100GB what we can do in 1TB today, 10x-ing our model size[1] from today won't be better?
[1]whether it is 10x in size, or maybe running 10x the models to have things like judges etc.
SoTA AI will move well past 1TB+ memory requirements in 1 year or so.
Also, you will be able to play with much more competent models locally. They will still feel like children compared to the adults living in the SoTA region.
And besides llm's memory requirements:
-HBM has a lot of room to grow(bandwidth/power/density). And an HBM memory takes 3x the amount of wafers of regular DRAM.
-HBM + better DRAM designs could increase random memory access speed by alot. I've seen research talking about 7x.
At what point it makes sense that half of consumer GPUs and most of the server GPU/CPUs start using HBM?
Are you thinking of fundamental architectural changes (like the Transformer)? Or incremental (like MoE or GQA)? Are there specific neolabs or techs you are following that lead you to this prediction?
Indeed it seems quite possible we are one architecture breakthrough away from existing chip stock driving us all the way to ASI.
Nobody has done AFAIK the information theory to prove it’s not possible.
Yeah, you can directly print the SOTA AI model on the chip. I also believe this can be done on older process nodes. The most important factor would be speed. How fast can you go from new model to new chip?
Even if we do, we are going to need a lot of RAM for the billions of agents running everywhere.
"we" being China?
I'm looking forward to being able to buy CXMT chips for running SOTA models locally. :)
You can get _better_ that SOTA AI performance by burning the weights directly to silicon, the problem is you're locked into that model
NOPE. The opposite will happen. CXMT will flood the market thus making 1 TB models affordable.
that would be perfect timing
Hyperscalers are screwed, data center mania couldn’t even be completed during this massive spending spree, all while the people seemingly sitting on the sidelines are working on getting these things baked into the OS and chipsets in a way consumers wouldn’t notice
[dead]