The weights are mapped directly to transistors. It's not a generic processor, it's literally a dedicated Llama 8B chip that can't be used for anything else. When you specialize in hardware you get faster - Taalas is pushing that to the limit.
They seem to be doing well. I checked recently and their API is closed to signups due to overwhelming demand.
Woah. How is this working? It's stupid fast.
The weights are mapped directly to transistors. It's not a generic processor, it's literally a dedicated Llama 8B chip that can't be used for anything else. When you specialize in hardware you get faster - Taalas is pushing that to the limit.
They seem to be doing well. I checked recently and their API is closed to signups due to overwhelming demand.