← Back to context

Comment by rhdunn

1 hour ago

From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:

    f_gate(b,x) = b * tanh(x / b) * sigmoid(x)

    f_up(b,x) = b * tanh(x / b)

Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).

2 comments

rhdunn

Reply
lostmsu  17 minutes ago

Which part of that is doing gating? I thought gate generally looks like

    f_gate(x, y) = f(x) * y

for some f. In your case b is a constant hyperparameter though, e.g.

    f_gate(x) = 4 * tanh(x / 4) * sigmoid(x)

So where's the gate?