Comment by YetAnotherNick

Depends on if you are using tensor parallelism or pipeline parallelism, in the second case you don't need any sharing.