Comment by jedberg
5 months ago
I've been in the security space for 25 years, and understand the theory of PKI. But I've also been in the ops space for 30 years, and understand that if you don't balance security theory with operational practice, critical business functions can fail.
Ideally yes, the private key is never seen. In reality, it needs to be backed up in a secure place so it can be restored in the event of a failure.
You can use more than one key you know.
Keep the private key you actively use in the secure enclave. The system you actively use is most at risk.
Keep a secondary offline private key as backup. You can generate and store it in a secure location, and never move it around. Airgapped even if you want. You could even use a yubikey or other hardware for the secondary key giving you two hard to export keys.
Distribute pub keys for both of them.
Best of both worlds?
It’s important to remember that over time systems develop complexities that can be hard to recover from scratch because by definition air gapped data aren’t ones you are regularly exercising. Here’s an example of this in action from Google’s history
https://google.github.io/building-secure-and-reliable-system...
>It took an additional hour for the team to realize that the green light on the smart card reader did not, in fact, indicate that the card had been inserted correctly. When the engineers flipped the card over, the service restarted and the outage ended.
This is some good read!
Yeah but if you get a new device, you have to go add its pubkey to every server you ever use. I wish there were an easier way, otherwise it's understandable that people copy privkeys.
There is an easier way: Create a SSH CA, add that to your authorized_keys everywhere, use it to sign the individual public keys.
6 replies →
> if you get a new device, you have to go add its pubkey to every server you ever use
It’s not too bad, if the number of servers is not too high.
I have different client pub keys on my phone, multiple laptops and desktop computers and manage my authorized keys to be able to ssh into my servers from the devices, as well as from one laptop to another or from my phone to one of the laptops, etc.
Because I already have several client devices I don’t really need any backup ssh keys. The fact that each device has a different key means that if one laptop breaks or my phone is stolen, I can still ssh into everything from one of the remaining devices and remove the pub key of the broken or stolen device from authorized keys and generate new keys on new devices and then using one of the existing devices to add the pub key of the new device to the authorized keys of the servers and other devices.
For me it’s manageable to do it manually. But if you have very many servers you’d probably want to use a configuration management tool like Chef, Ansible, Puppet or Saltstack. Presumably if you have a very high number of servers you’d already be using a configuration management tool like one of those for other configs and setup anyways.
There is an easier way, it's called TLS certificates, it's just that SSH decided not to use it for some reason.
Other systems of this nature have figured out long ago that you should be able to have one personal certificate (stored securely in an airgapped environment), from which you'd generate leaf certificates for your devices every year.
3 replies →
If you’re operating at the scale this is too cumbersome to do manually surely you already have a configuration management system in place to automate this no?
1 reply →
> I wish there were an easier way,
For SSH there is.
Its called SSH certificates. ;)
That can work really well for systems where you don't need to share your key material very often, or where sharing is optimized for n-key scenarios.
SSH isn't always that. For example, ssh-copy-id by default does not copy over multiple identities.
For that reason, I'd personally prefer to import my (otherwise airgapped) key into my secure hardware exactly once and mark it as non-exportable in the SSH scenario.
I think you are mixing up concerns. You need a backup key. That doesn't mean you need to backup your key.
Anything (everything?) using SSH authentication supports multiple authentication keys. Have a yubikey in a locked deposit box or something.
Today I make a private/public keypair, and the private key is on my laptop in my encrypted home folder. It also gets backed up to my encrypted offsite backup. That way if my laptop breaks or is stolen, I can restore from backup and be up and running as before.
I was simply asking if that is still possible with this method, nothing more.
And not every service that uses ssh auth allows multiple keys.
It’s possible but would not bring you any extra security.
The advantage of non-exportable, HSM-backed keys is that you are guaranteed that the only way to use that key is to have online access to the HSM, and you can recover from HSM access compromise without having to replace the keys.
If you make the key exportable it is no better than if it was stored on your disk to begin with.
yes, you can export keys using this method, and they will be simlarly secure as password encrypted keys you generate without the secure enclave with openssh, but with the convenience that you can decrypt the key using TouchID on macOS.
Such a setup is marginally more secure than just typing in the passwords, since it is much harder to intercept the TouchID chain from touch to decrypting the SSH key compared to your keyboard to the terminal.
All that said, here are the priorities of a few security technologies:
TouchID:
1password:
I cannot tell you really what is "#1" in security among packaged ready to buy commercial products, Everyone, practically, makes affordances for convenience ahead of security. I suppose there isn't really a great product for normal people that puts security first. Of course, there are an ad hoc collection of practices that amount to, #1 security. But a product? No. Even Apple Lockdown mode... well, they can still just push an update that makes it pretend it is enabled when it is not, so...
As others mention, there is no point to using the Secure Enclave if you have your key stored on disk or in your backup. It’s like putting impressive locks on the front door, while leaving the window open.
Beyond that, you can do that just fine right now by making TWO keys. If you lose the laptop, oh well. Recover with your backup key (which is hopefully kept more securely than you describe - it can be inconvenient to access since it is only needed for recovery).
This also lets you go further in locking things down or providing you notifications, as you can distinguish server side between your usual key and the backup key.
The point of the enclave is to be noncloneable and access limited. Extracting the key for the backup would negate the benefits derived from that.
1 reply →
If you are keeping it on disk (encrypted or not), you don’t really gain anything from using the Secure Enclave.
No but that’s the point. If there’s a copy of the private key out there, then it can be copied. The whole point is that the jedberg-laptop-1 key only ever exists as jedberg-laptop-1. When that laptop gets lost/stolen/destroyed/aged out, there's 100% certainty that it can't be recreated. The other side of that equation is that you need a tree of keys and a whole IT department to manage them so you don't get locked out of your servers. This particular bit of software is about ssh keys and exists within a larger conversation about PKI which you know more about than I, but operationally, you have this, and then you have a root login private key file locked with Shamir secret sharing (ssss on debian) that you distribute to a very select few number of key bearers. And then don't all get on the same plane together, ever.
This was a thing with Google Authenticator. People kept asking how to back up or transfer keys, official answer was you can't and WAI. Eventually they conceded and added a backup option, but it was still confusing. I think this ruined the entire reputation of TOTP.
> if you don't balance security theory with operational practice, critical business functions can fail
i.e. people will circumvent the secure-but-onerous path. (I don't think they can be faulted for trying to get their work done either, I'm agreeing with you)
In what scenario would you prefer to backup an SSH key in favor of generating new SSH keys?
When I have my pub key in the authorized_keys files of many machines, especially machines where I don't control the authorized_keys file.
In this case you can maintain an offline SSH CA and trust that on the remote machines, and then sign yourself leaf certificates against a non-exportable HSM-backed key. In case of loss you just make a new key and sign a new certificate.
Of course this just moves the key management problem somewhere else: now you need to protect the CA key, but that might be easier since you would only need access to it in a disaster recovery scenario if you replaced the laptop or otherwise lost access to your HSM-backed key.
As usual, it all depends on your threat model.
4 replies →