Comment by chorizo
6 hours ago
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
We also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
Baking the base models on to ROM makes a lot of economic sense.
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
This stopped working some years ago.
Now you sell the same phone with higher price tag.
I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I would guess.
Ofc if the model has some critical bugs that’s another matter.
Your examples worked on phones for over a decade.
Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
And the customers can wait for the new phone released next year. These are edge models - the average customer doesn’t need the latest frontier model. Just needs to be good enough for the features you promised.
That's a software engineering problem. They just need to figure out how to fine-tune for alignment and tool usage.
That's the only thing the normie consumer cares for really.
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
Did you try using the the talaas chat? Something stupid like 18k tokens/second.
Think it's called Askjimmy or similar.
Oh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
I thought DRAM was pretty dense already. Is mask ROM that much denser?
2 replies →