Comment by jcase

With SSH you usually own both endpoints (or at least trusting your cloud provider).

The example you give with regards to exchanging a piece of paper is very similar. It's ridiculously hard to do such a thing on large scale without trusting intermediaries.

I'm putting my eggs on certificate pinning.