Comment by Melatonic
9 hours ago
The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )
9 hours ago
The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
AMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B.
Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others.
[1]https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
Baking the base models on to ROM makes a lot of economic sense.
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
1 reply →
I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I would guess.
Ofc if the model has some critical bugs that’s another matter.
1 reply →
And the customers can wait for the new phone released next year. These are edge models - the average customer doesn’t need the latest frontier model. Just needs to be good enough for the features you promised.
That's a software engineering problem. They just need to figure out how to fine-tune for alignment and tool usage.
That's the only thing the normie consumer cares for really.
We also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
Did you try using the the talaas chat? Something stupid like 18k tokens/second.
Think it's called Askjimmy or similar.
1 reply →
The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
3 replies →
Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.
Its a cool demo, but its gpt-3.5 level stupid, or worse.
edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.
It's a quantised 8B model (Llama 3.1 8B to be exact).
[1] https://taalas.com/the-path-to-ubiquitous-ai/