Comment by inigyou
14 days ago
I've never heard of any reliability reason you shouldn't run checkpoints whenever you want - only performance reasons. Can you elaborate?
Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API.
But you don't need checkpoints for consistency and I think the backup API will not copy both the old and new versions of pages just because they're in the WAL, in other words I think checkpointing then backing up should give you the same pages as backing up without checkpointing. So the whole thing seems unnecessary.
My generalist reading of this (which is applicable in lot of cases) is that a dedicated flow is handling the house keeping part of the job, aka resetting the checkpoint after writing the WAL to disk. These one off house keepers are common in a lot of softwares and they expect to work alone, they are tested to work alone. They always have a set of ritual, rules and order in which they do all the things. Now if some other thread takes off some of those jobs then they break this routine for the house keeping job and this new thread/person may not always know what else has to be done before and after this one particular job for the sake of completeness.
On second question of why the tailscale developers did it, its possibly for the same reason why they invested this much into debugging this issue. Some one believed the current behavior did not fit into their architecture, they want to be more performant and take control over things. A big part of me considers this is a required exercise to try, grow and learn. The only thing they could have for improvement would be to have these old hands on architect kind of folks on their team who might have hinted/pointed them to the problem a long before. Challenge/chances are that these older folks would have even stopped them from going in this direction in the design phase itself.