Comment by Dagger2

Random guess: PMTUD? Like on v4, some people fuck up their PMTUD and are incapable of realizing or fixing it, so you have to have some kind of workaround.

If setting your client machine MTU to 1280 (`ip link set mtu 1280 dev eth0` or equivalent) magically fixes it, that's your problem.