Comment by inigyou
14 days ago
This reaffirms my belief that SQLite is not well suited for systems with significant concurrency. It replaces fopen, not postgres. Although this corruption is a rare bug and sqlite is usually extremely stable, it's usually not worth it from a performance and features standpoint either.
Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online continuous backups.
Eeeh, this particular bug was a race when applying WAL to the data files in a checkpoint. Postgres’s checkpointing is theoretically just as vulnerable to this class of bugs as SQLite, though it doesn’t seem to have equivalent issues today.
Online backups/replicas are nice until you’re charged for network traffic or have to recreate the replica from scratch, at which point the initial-restore-then-hook-up-the-WAL-stream dance is prone to all sorts of racy issues. If you’re lucky enough to have only a single process talking to the DB, SQLite seems like a nice way to sidestep that complexity while keeping a simple backup story.
Heck, this is basically the Redis model: a single process/thread coordinates all access to the data, and occasionally forks off a background job to snapshot the state somewhere. From that perspective, the Tailscale controller binary is a database; SQLite is just the data file format.
Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear.
It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple of years... Or you could just immediately go with a solid Postgres (or Postgres-like setup, eg. yugabyte) and save yourself a bunch of faffing about with weird solutions and migrating between them. But I guess that doesn't drive engagement on your blog.
Postgres has also had pretty serious bugs, eg fsyncgate
There's also the more fundamental issue that Postgres - unlike something like Yugabyte - does not use a distributed consensus algorithm for writes and can lose committed writes during network partitions.
But of course, neither does Tailscale's weird DIY contraption.