Comment by warkdarrior
8 hours ago
> Burning the weights into the memory with local memory cores capable of the kernel operations would be a lot more efficient than round tripping busses.
Sure, but now we're not talking about just burning the weights into the chip, but also designing a new architecture that has memory local to each core. A new architecture would then require a new programming model, which means new inference stack, which may mean new training stack.
I don’t think it would require a new training stack, and I’d imagine it makes more sense to distribute cores with memory. The cores can be simplified to the functions of the kernel since the inference kernel can be expressed as a reduced set of optimized functions in the pipeline rather than a general CUDA core. If the model is burned into ROM, the compute pipeline can be baked into the core.
Burning the weights into static mask ROM is pretty trivial. Taalas is doing it.