Comment by lostmsu

Which part of that is doing gating? I thought gate generally looks like

    f_gate(x, y) = f(x) * y

for some f. In your case b is a constant hyperparameter though, e.g.

    f_gate(x) = 4 * tanh(x / 4) * sigmoid(x)

So where's the gate?