Comment by imtringued
2 hours ago
They advertised it as 10% to 15% worse than transformers
Not sure what the point of the paper's title is if the resulting model does not match the transformer.
2 hours ago
They advertised it as 10% to 15% worse than transformers
Not sure what the point of the paper's title is if the resulting model does not match the transformer.
Well they're quite explicit about that:
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.