Comment by imtringued

The reason why transformers are the best architecture has to do with the fact that tokenization introduces a variable quantity of manipulable objects with cross interactions.

You can think of each token as a semantic database row with semantic properties (color, shape, etc) associated with said row. Classic NNs have to waste neurons on composite representations like "dark circle" because if they had just color or similar properties they run into the issue that e.g. a classification must classify the color of the entire image rather than the color of an object inside the scene. To get around that limitation, the network has to then implicitly tie concepts like color into the depicted object (dark circle) so that it retains the ability to distinguish the objects from the property the object has.

What a transformer does is basically reuse the same network over and over again on each abstract semantic database row but not just in the classic RNN fashion where you only have the current state. The transformer calculates the token by taking every other token into account. The K matrix forms a classifier network on the fly that detects the relevancy of a token pair and the V matrix forms a regressor network that takes the token pair and produces the expected output of the combination of the two tokens. When you combine the relevancy plus the regressor output you get a weighted sum, essentially emulating an arbitrary function that has read the past input (like a turing machine).

Edit: This also means that you should bet against single state systems like Mamba. The system needs to be able to deal with multiple parallel states and recombine them (e.g. via attention). Single state Mamba is dead! Long live parallel state Mamba!

Edit 2: We should also start questioning the concept of a token altogether since it is equivalent to an inflexible flat database row. What we really need is the concept of a token group where one part of the token refers to the key and the other part of the token refers to the value for that key. This would give each token the ability to encode an arbitrary JSON style document or an arbitrary self contained graph.