Comment by xyzzy_plugh
2 days ago
This never should have been on by default. The end user (read: administrator) needs to know they want to use the TPM.
This is a huge foot gun for many devices.
The accompanying changelog note hints at why:
> Failure to load hardware attestation keys no longer prevents the client from starting. This could happen when the TPM device is reset or replaced.
This is unfortunate as for many, many deployments, you absolutely want this on. But because it's a time bomb for certain device/OS combinations that Tailscale can't control or predict, and folks install Tailscale on pretty much everything, then the incidence of borked installs can only rise.
As someone with a passing interest in using TPM for crypto things, everytime I think deeply about the implementation details like this, I come back to needing some kind of recovery password/backup key setup that entirely negates the point of the TPM in the first place. They seem really neat, but I struggle to see the benefit they have for doing crypto when a tiny slip up means your users' keys are poof, gone. And the tiny slip up may not even be with your software, but some edge case in the OS/TPM stack.
The TPM was never designed to be the only holder of a key that cannot be reset. The idea was that it prevents you from typing in a password or reseting an attestation signature in a database for 99% of boots, but if certain things in the boot process change (as determined by the firmware, the CPU, the OS, and the application using the TPM) it’s designed to lock you out so those things cannot change without anyone’s notice.
For that purpose they’re pretty good, though there are advantages to a more signature-oriented boot security option like Apple’s Secure Enclave. But that only works so well because Apple simply doesn’t permit altering parts of the macOS boot process. For Windows/Linux, you have a variety of hardware, firmware, and OS vendors all in the mix and agreeing on escrow of keys for all of them is hard.
The presumption is that the contents being secured are /so/ valuable that locking my device is preferable to any leak of them whatsoever.
This is military level security and just isn't appropriate for most consumers. Particularly around something so rarely exercised and utilized by users as the boot process. A simple warning with a long timeout would have sufficed.
Aside from that you have a hardware vendor, sourced into an integrated product from another vendor, sold to a user, with various third party software interacting with it. This was always going to result in questionable experiences for end users.
3 replies →
Whether by design or accident, this is correct.
You backup a key or key creation mechanism or whatever elsewhere somewhere very safe.
Then almost never touch it, as the TPM authenticates.
The primary argument in favor of TPM's is the desire to assert against tampering to the boot system, and as a secondary effect it can be one of the solutions to reduce the need for users to type in passwords.
You can still use crypto without a TPM, including with full disk encryption, and for LUKS specifically you can use multiple passwords and mechanisms to unlock the system. Different solutions will give different benefits and drawbacks. Me and a friend wrote a remote password provider for Debian called Mandos which uses machines on the local network as a way for unattended boots. It does not address the issue of tampering with the bios/boot loader, but for the primary purpose of protecting against someone stealing the server/disks it serves the purpose of allowing us to use encrypted disk without drawbacks of typing in passwords, and the backup server, itself with encrypted disks, handles the risk of needing recovery passwords. At most one needs to have an additional backup key installed for the backup server.
TPM keys are great for things like SSH keys or Passkeys, which surprisingly works well even in Windows.
The private key is safe from any exfiltration, and usage only requires a short PIN instead of a long passphrase. The TPM ensures you're physically typing that PIN at the machine not a remote desktop window or other redirection that could be hacked.
Obviously, this is problematic/annoying for scripts and things that can't share the SSH session, because you need to PIN with every authentication. Also, for encryption, you want to use something where you can backup the private key before stashing it in the TPM. Windows allows you to do this with certificates that are exported for backup prior to encrypting the private key with an unexportable TPM key in Hello.
An easy solution to having to put your PIN in too often for SSH is to use the `ControlPersist` option in your SSH client config. This lets you only create a new SSH connection every 30s (or whatever you put), even if you’re doing separate operations. With a low timeout, there’s no realistic security risk (what’s the chance an attacker will only have control of your machine for 30s?).
I do this for GitHub in particular, because of tools that connect to the remote multiple times. Works with anything that uses the actual ssh executable under the hood.
Same with passkeys actually.
Passkeys get synced between your devices so they aren't any more fragile than passwords in a password manager.
6 replies →
The benefit is that you don't enter the recovery password most of the time.
And when you do it should be rare and lead to a password reset.
But e.g. Windows uses a TPM by default now ? If TPMs were such a major issue then there would be millions of Windows users with TPM problems, no ?
I have no inside info, but this strikes me more as a bit of a "sledgehammer to crack a nut". Tailscale turning off important functionality due to small-but-vocal number of TPM edge cases ?
It is also very unfortunate they did not manage to find any middle ground between the hard-binary all-on or all-off.
Windows uses TPM for Bitlocker. A very common scenario where TPMs get reset is BIOS updates (when a TPM is implemented in firmware). AFAIK, Windows cheats here because it also manages BIOS updates. When an update happens, it takes extra steps to preserve the Bitlocker encryption key in plaintext, and re-seals it to the TPM after the update completes.
Apart from Windows, there are many setups that fail in fun ways: Kubernetes pods that migrate from one VM with a TPM to another one, hypervisors that mount a virtual TPM to VMs, containers or VM images that do Tailscale registration on one machine and then get replicated to others, etc.
Tailscale already did some attempts at cleverness when deciding whether to enable features using a TPM (e.g. probing for TPM health/version on startup, disabling node state encryption on Kubernetes pods), but there was still a long tail of edge cases.
> Bitlocker encryption key in plaintext
Actually, this is not the case. BitLocker wraps the key, meaning even if the TPM were compromised, one would still have to brute-force the PIN for the actual key. It’s cryptsetup on Linux that stores the key on the TPM in plaintext. This vulnerability has been known for quite a while and nothing has been done about it so far.
https://arxiv.org/abs/2304.14717
https://github.com/systemd/systemd/issues/37386
https://github.com/systemd/systemd/pull/27502
> Windows cheats here
Slightly off-topic: it also cheats in how TPM works for Bitlocker when you do TPM + PIN. One would assume PIN becomes part of the encryption key, but in reality, it's just used as the auth for TPM to release the key. So while it sounds like a two-factor solution, in reality it's just single factor.
So the Bitlocker without TPM is actually a better idea and Windows makes it very painful to do if TPM is on.
4 replies →
> Windows cheats here because it also manages BIOS updates
Is this (relatively) new?
I don't use TPM and I rarely update BIOS unless I really need to, but I thought there was an option on my BIOS/UEFI to use USB drive to update it. How would Windows know about it?
4 replies →
On Windows, certificates can also be stored in the TPM.
Windows seems to do two big things with a TPM. Bitlocker encryption and some microsoft account stuff.
If the bitlocker stuff goes wrong, big problem, hopefully you printed and kept your recovery key.
If the microsoft account stuff goes wrong, mostly the microsoft store and microsoft store apps break in subtle ways... but that's also how that ecosystem normally works, so how are you supposed to know it's the TPM problem?
Windows automatically reinitializes the TPM if it's reset boots normally, most end users will not notice any issues unless they have Bitlocker or biometrics configured.
The problem here seems to mostly have been that some exotic virtualization software insists on offering broken TPM.
>> This could happen when the TPM device is reset or replaced.
Isn’t that exactly the desired behavior to defend against physical attacks?
Sure, but most users probably don't actually want this level of defense.
For the same reason that most folks don't use bank vault doors on their house.
Ex - even reasonably technical people hit this footgun in lots of edge cases... like updating their bios, changing the host of a vm running the tool, or having a k8s pod get scheduled on a different node.
I'm surprised this was "default on" at all.
Yes, but it turns out the TPM gets reset quite often on shitty hardware.