Comment by Dagger2

That will only give the NTP server the IP you use for outbound connections. If you use privacy extensions, they'll get a temporary address.

If you don't configure your firewall to allow inbound connections to the temporary address, knowing your temporary address doesn't help them connect back to you. (Also, it's temporary, so their logs of IPs will be useless after a small window.)

Compare this to v4, where connecting out to someone gives them enough information to exhaustively port scan your whole network and trivially find every server you're running.