Comment by Dagger2

I could do it if it was using a routable v4 address too, and I can do it with either RFC1918 or ULA as well (which are both routable, just not over the Internet) if I can get close enough to send the relevant packets. NAT provides no protection against any of these.

You don't normally see many SSH brute force attempts on v6, let alone getting hammered by them. I do see some, but it's mostly to obvious addresses like <prefix>::2, ::3 etc which I don't use, or to IPs you can scrape from TLS cert logs. If you set an ssh server up on an IP that you don't publicize, finding it is hard.