Comment by vidarh

You're assuming someone would be running Emacs on the remote machine talking to a local X server in order to edit files on a remote machine, but people would generally not do that, but use something like TRAMP, where Emacs would be running on your local machine, but accessing remote files.

TRAMP only requires ssh or telnet (or scp, rsync, any number of other methods) on the remote machine.