Comment by throwaway984393
4 years ago
It's a trade-off. If you have no disk, the disk can't fail, but the network can, and the remote PXE server can, and the remote SAN can. You can get into a state where you have to pray no servers reboot. Intermittent errors can be real annoying when it makes provisioning fail. (used to work a server farm that'd do server rebuilds over PXE, and ran a few diskless cluster projects)
An alternative is you use a RAID array and mount your disks in read-only mode, or use physically read-only disks and when you have to replace a disk, you pre-mirror the replacement disk. In this way the local disks can be replaced as they fail and there's never a point when the server is at risk of not being able to boot.
......or they could boot from CDROM :)
A network or PXE server can fail regardless, so this are things that always have to be taken into consideration and in those instances then you address those issues. With this type of setup you do not need a remote SAN as it would defeat the purpose of not having external storage that could store logs. Mullvad has servers all over the world, so a temporary failure in one location will not bring down their entire infrastructure.
It's not just a temporary failure, it's potentially the entire AZ going down hard. High Availability network boot without local storage is very difficult/expensive.
They can still use local disks to provision the OS over a network but boot from local storage, and prevent writing to disks from the booted OS (hell, they can completely remove the disk drivers from the kernel!). It just doesn't make sense to ditch the drives from a reliability standpoint. They're going to have a big outage one day just because they didn't want to deal with drives.
Mullvad and similar providers often colocate or rent servers from multiple local hosting providers. A group of servers going down for them would not be a big deal. Network boot is not difficult/expensive. Many of their servers are using 10Gbit uplinks+ so I take it they get pretty good deals for bandwidth. It isn't like Amazon or other cloud providers that charge an arm/leg for egress.
The point of not using local disks is again fairly straightforward, to show that they do not have a stateful storage medium to write logs to. Whether it significantly helps or not is beside the point, they have determined that it helps provide assurance to their customers and additionally showcases a feature for auditors.
Network booting loads the OS into RAM, so even if there was a network outage they'd have to restart the severs to cause a problem. From what I know of most VPN solutions though, again a network outage would only affect the group of servers at that data center which isn't their entire operation.