Comment by jolmg
5 years ago
> They ensure you don't get a garbled filesystem.
Well, they do that, but they also protect data to reasonable degrees. For example, ext3/4's default journaling mode "ordered" protects against corruption when appending data or creating new files. It admittedly doesn't protect when doing direct overwrites (journaling mode "journal" does, however), but I'm pretty sure people generally avoid doing direct overwrites anyway, and instead write to a new file and rename over the old one.
I'm not sure if it would protect files that are clobbered with O_TRUNC on opening (like when using > in the shell). I would imagine that using O_TRUNC causes new blocks to be used and so the old data isn't overwritten and it isn't discarded because the old file metadata which would identify the old blocks corresponding to the file would be backed up in the journal.
> They also expose an API that allows you, if you're very careful and really know what you're doing (like danluu or the SQLite author), to write performant code that won't garble files on random shutdowns.
As far as I see for the general case, being "very careful and really knowing what you're doing" consists of just avoiding direct overwrites. Of course, a single file that persists data by the needs of software similar to a web server (small updates to a big file in a long-running process) is going to want the performance benefits of direct overwrites. I can totally see SQLite needing special care. However, I don't think those needs apply to all applications.
When I said "allows you, if you're very careful and really know what you're doing, to write performant code that won't garble files", by "performant" I was alluding to direct overwrites. If you don't need direct overwrites (perhaps because your savefiles are tiny), then no problem. If you do, you should use SQLite or LMDB or something, unless you work at Oracle or somewhere else where your job is to compete with them.
The example I had in mind was Word, which gave up on direct overwrites and managing essentially their own filesystem-in-a-file in favor of zipped XML, which is really good enough when writing a three-page letter, but terrible when writing a book like my mother is. Had they used SQLite as a file format, we would've gotten orders-of-magnitude faster save on software billions of people use every day.
> As far as I see for the general case, being "very careful and really knowing what you're doing" consists of just avoiding direct overwrites.
That's a dangerous thing to say. There are many ways to mess up your data, without directly overwriting old data.
If you write a new file, close, then rename, on a typical linux filesystem, mounted with reasonable options, on compliant hardware, I think you should have either the old or new version of the file on power loss, even if you don't sync the proper things in proper order, but that's only because of special handling of the common pattern. See e.g. xfs 0 size file debacle.
Not an expert.