Comment by traceroute66

> I wish there were an easier way,

For SSH there is.

Its called SSH certificates. ;)