Comment by danpalmer
15 days ago
Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.
15 days ago
Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.
Maybe, but there’s nothing about this particular bug that’s due to SQLite not being a networked database. Postgres is just as likely to have TOCTOU races in its checkpointer, which works roughly the same way.
Tailscale likely deals with a lot of security-sensitive traffic, given the nature of the service. I'm guessing one of the requirements they were given was to have a tiny blast radius in case encryption keys got leaked, and that meant isolating each customer's tailnet (meta)data to it's own sqlite db rather than letting everyone share a postgres cluster.
How about having devices generate and not share their private keys, instead?
That's already how it works at the data-plane layer. I'm talking about the control-plane layer where tailscale needs to maintain a server that gets its state from somewhere (they need to know where to route your data, what your permissions ACLs are, device names, etc.). If everyone shared the same db cluster accessed over the network there is a real risk of an accident leaking the encryption keys (of the database itself, not your tailnet) to the db they would have to maintain, and leaking semi-sensitive info like these device names and IP addresses. Using sqlite means each tailnet's metadata is isolated to the container running it.
3 replies →
Yeah it’s a pity that wasn’t addressed in the article.
This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”
That assumes that they do actually believe it's a mistake. They didn't explain the reasons they've gone for this architecture in much/any detail. I'd be interested in hearing them talk more about that in the future.
Perhaps you or I would make a different decision based on the aims that lead them there. But there isn't enough information to say whether or not their decision was a mistake, even if it has lead to a peculiar bug. It may be perfectly legitimate and we just don't know some of the constraints they had.
Right, the point is it's a kind of elephant in the room - they even allude to it not being a common use case - but they say nothing about the rationale. That leads me to suspect it was a kind of "seemed like a good idea at the time" situation.
They also touch on this when they say that it worked for them for a long time.
This is a pretty classic symptom of a system that was designed a certain way early on and then runs into issues as the system grows.
One can argue that this was due to a bug, but it's a bug that they shouldn't really have had to deal with - a consequence that the design opened them up to.