That's exactly why it's called a 'rectified' linear unit! It's a half-wave rectifier. The ReLU function is just what you'd see if you put an (ideal) diode on a curve tracer.
(Although to be more accurate, it would be what you'd see on an I-V curve tracer if you measured an ideal diode in series with a 1-ohm resistor. The diode by itself would just go vertical at the Y axis, and the ML people would mutter into their beards about exploding gradients.)
That's exactly why it's called a 'rectified' linear unit! It's a half-wave rectifier. The ReLU function is just what you'd see if you put an (ideal) diode on a curve tracer.
(Although to be more accurate, it would be what you'd see on an I-V curve tracer if you measured an ideal diode in series with a 1-ohm resistor. The diode by itself would just go vertical at the Y axis, and the ML people would mutter into their beards about exploding gradients.)
And that diode would be an exploding grenadiant?
1 reply →