Comment by Melatonic
7 hours ago
The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )
7 hours ago
The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
Baking the base models on to ROM makes a lot of economic sense.
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
3 replies →
Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.