Interesting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?
Taalas only uses SRAM for the KV cache and the activations, the weights are in mask rom in the metal layers.
If they designed this right, it means that once they have a model, so long as they keep the hyperparameters fixed they can change the weights much faster than it takes to spin up a completely new chip, essentially at a cost of doing a minor revision.
Interesting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?
The weights are in SRAM, so the LLM architecture is burned in but the weights can be updated.
Taalas only uses SRAM for the KV cache and the activations, the weights are in mask rom in the metal layers.
If they designed this right, it means that once they have a model, so long as they keep the hyperparameters fixed they can change the weights much faster than it takes to spin up a completely new chip, essentially at a cost of doing a minor revision.
2 replies →
No the weights are in the metal layers, they cannot be updated.
2 replies →
8B is still three orders smaller than current frontier though