Comment by scottlamb
7 years ago
You have to synchronously report the error to the application depending on the fsync, full stop. None of the other proposed mechanisms are a replacement for that; for one, because they are asynchronous. An application can't be written to "do an fsync, assume it's done...unless some system-wide error reporting mechanism says otherwise arbitrarily later".
> FWIW: I'm all but certain you're wrong about the "POSIX-violating" part. My read of the standard says the author was thinking only about the current request.
An fsync call is not a complete request (it doesn't specify the data to be written), so what on earth does "only about the current request" mean?
I can't reconcile any of these arguments that Linux's behavior was correct with the RATIONALE section here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... and particularly the phrase "all data up to the time of the fsync() call is recorded on the disk". There's no mention of "since a previous fsync() call" or any such thing there.
> You have to synchronously report the error to the application depending on the fsync, full stop.
Why? As reported above, Google itself isn't even doing that. We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work.
The fsync behavior on Linux was surprising and wrong. But fixing that won't fix any of the stuff you're yelling about. API hygiene can't fix machines, and the machine broke.
> Why? As reported above, Google itself isn't even doing that.
I work at "Google itself". Typically there's a single process responsible for all the disk writes of significance on the machine, possibly bypassing the in-kernel filesystem entirely in favor of direct block device access. That process's caller waits until three machines (each with its own battery backup, and frequently on a separate PDU from the others) have acked the write before proceeding. Also, that process's caller is Spanner, and those three machines are necessary to say a single replica has accepted the write. Typically there are three replicas in different clusters (often in separate cities) and two of them have to accept the write (in Paxos terminology) before Spanner tells the application it's complete.
Please don't take Google's production environment working successfully most of the time from the customer viewpoint as proof of anything about the Linux virtual filesystem layer or ext4. You can paper over a lot of brokenness when you have six independent battery-backed copies of the data in two cities before you depend on it.
> We're not talking about a network timeout here. Hardware failures are hardware failures and not in the general case recoverable by software no matter how strongly held the opinions are about how that software should work.
Software can't fix the broken hardware, and (without significant redundancy and cross-verification as Google does) can't correct for hardware simply lying. But correctly written software can absolutely change the likely failure modes when hardware reports error. And intermittent hardware failures happen a lot more than you might think anyway. Bugs that turn intermittent failure into permanent corruption are nasty.