Comment by DocTomoe

Sounds like a problem easily solved with fail2ban. Which keeps legitimate folks in, and offenders out - and also unbans after a set amount of time, to avoid dynamic IPs screwing over legitimate users permanently.