Comment by dboreham

15 days ago

Quick note that data corruption bugs that are impossible to reproduce are not uncommon (perhaps they're the norm). So some amount of head scratching trying to figure out a plausible scenario by which the system could get into the state represented by the smoking remains is often required. Then you attempt to force it into the supposed bad state by modifying code paths accordingly. So the approach used in this case is clever, but it's not particularly unusual in the world of data stores.

I agree it's tricky, but Tailscale had downtime by this bug and the SQLite Changelog downplayed it.

  • How is "Fix the WAL-reset database corruption bug" downplaying this? https://www.sqlite.org/changes.html

    • > The bug is a data race with tight timing constraints. It is unlikely to occur in common use. The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.

      I read that and it feels like some theoretical thing versus causing impactful outages for workloads shaped like tailscale’s.