Comment by oofbey
6 hours ago
They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
I think there's a middle ground between "attention is obsolete" and "this proves nothing"