Comment by mapontosevenths

7 hours ago

The author describes the model's pairing operation as:

(zt,zt+Delta) where Delta is a positive offset.

This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.

I don't know if that's just the paper, or if the code was also b0rk3d.

5 comments

mapontosevenths

Reply
magicalhippo  5 hours ago

That's not how I read it.

For each position t and offset Δ∈ such that t+Δ≤L, we form a pair (z_t,z_t+Δ).

For a given t, this produces up to m pairs:

(z_t,z_t+Δ_1),(z_t,z_t+Δ_2),….

We treat these as local neighborhoods at multiple scales. Note that the pairing is causal: we only pair t with positions strictly to its right (future), consistent with left-to-right language modeling.

So z_t is a token in the input, and z_t+Δ is some later token. Guess they could have flipped that around to (z_t-Δ, z_t) with t+Δ>=0.

  • mapontosevenths  5 hours ago

    > So z_t is a token in the input, and z_t+Δ is some later token.

    I think you get what I'm saying, but when they compute the gate they do:

      h_t_new = alpha_t * h_t + (1 - alpha_t) * g_t

    If I'm reading it right g_t = geometric_mix(z_t, z_(t+delta))

    Which would mean that g_t includes t+Δ, which is a future token. It's leaking future tokens into the current math.

    So it could just be a sign error and still be right in the code, or I could just be getting it wrong.

    If I'm not though, and given that they got worse results than a standard transformer I really hope it's fixed in the code. If they do worse even when knowing the future it's bad news for the idea.