Comment by marcyb5st

You don't need memory because the output of the previous layer flows directly into the input gates of the next. So no I/O to/from HBM or similar

You still need some memory for the context, in flight answers, ... but not for the model weights and for the output of the intermediate layers. I found taalas demo here: https://chatjimmy.ai/