Comment by imtringued

It's not "not quite true", it's literally true because alternative architectures like RNNs and Mamba fully update their own internal states, whereas transformers only append to the context.

RNNs and Mamaba do not update their weights, but you could hypothetically scale the internal state to be as big as Fable's and GPT 6's parameters.