Comment by klaas-
14 days ago
> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.
I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)
We have very good reasons for our checkpointing model, related to our backup + disaster recover strategy, along with resource cost. It might be worth writing about one day, so I'll not give away all the details, but in very short form, we organize a backup strategy that has minimal pause time, avoids doubling the page cache cost of the database, and enables extremely fast byte-copy restores in disaster recovery.
okay but explain why you are using sqlite and copying the file to S3 instead of using any client/server DB and its online backup feature?
SQLite has an online backup API as well, but it is slower and requires a significant additional page cache cost.
The team chose SQLite early on (there are some blog posts about this) and then we vertically scaled against the SQLite architecture. There are subtle ways you come to depend on the proximity/latency when you scale with local storage that mean switching requires a lot of non-obvious work - it’s probably the largest hazard for embracing SQLite in a growing saas - but at the same time you can push the vertical scale pretty far, which has great margins.
Had we scaled a different architecture of database there’s little reason to believe it would have been plain sailing as seems to be implied here.
I don’t think that’s an architectural solution to this class of bug. I replied to a similar comment here: https://news.ycombinator.com/item?id=49281533
This reminds me why RFC 2119 includes the description it has regarding SHOULD.
Like this is a great practical takeaway, sure, but eh.