Comment by gus_massa

That's 100G to 1T. Neurons are more complicated than weights, so add a x1000 somewhere, but they have more redundancy because they are noisy, unrelilable and may even die, so divide by some small number.