Comment by imtringued

No my point is more general. The headline promises me that I don't need attention based transformers, then you see the benchmarks and they tell you they got 10% worse perplexity. How is that going to convince me that I don't need attention?

I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.

If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.

[0] https://arxiv.org/abs/2603.15569