Comment by saagarjha

There are plenty of matrix multiplies in the backward pass too. Obviously this is less useful when serving but it's useful for training.