Comment by CamperBob2
3 days ago
That's exactly why it's called a 'rectified' linear unit! It's a half-wave rectifier. The ReLU function is just what you'd see if you put an (ideal) diode on a curve tracer.
3 days ago
That's exactly why it's called a 'rectified' linear unit! It's a half-wave rectifier. The ReLU function is just what you'd see if you put an (ideal) diode on a curve tracer.
(Although to be more accurate, it would be what you'd see on an I-V curve tracer if you measured an ideal diode in series with a 1-ohm resistor. The diode by itself would just go vertical at the Y axis, and the ML people would mutter into their beards about exploding gradients.)
And that diode would be an exploding grenadiant?
Yes, because the slope of current versus voltage goes infinite as soon as the voltage goes positive. The math would explode, and so would the diode, given enough current.