Comment by mapontosevenths
4 hours ago
> So z_t is a token in the input, and z_t+Δ is some later token.
I think you get what I'm saying, but when they compute the gate they do:
h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t
If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))
Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.
So it could just be a sign error and still be right in the code, or I could just be getting it wrong.
If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.
Fair point. If only the paper had a link to the code.
I found this[1] repo which tries to reproduce it, and its accompanying blog post[2] uses (z_t, z_t - delta), without further mention.
Another repo[3] trying to reproduce it also does the reverse from what I can see, so (current, past).
Interestingly the first repo got underwhelming results, while the second repo matched the paper roughly. Not sure if it's related to the Claude-usage or not in the first, though it notes a lot of details were left out of the paper which is true.
[1]: https://github.com/Infatoshi/grassmann-flows/
[2]: https://github.com/Infatoshi/grassmann-flows/blob/main/blog....
[3]: https://github.com/kanenorman/grassmann
They advertised it as 10% to 15% worse than transformers
Not sure what the point of the paper's title is if the resulting model does not match the transformer.