Comment by hedora
7 years ago
Every computer component can fail in arbitrary ways, including drives.
If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data.
Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack.
Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process.
It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible.
From https://lwn.net/Articles/752063/: "Linux is not unique in behaving this way; OpenBSD and NetBSD can also fail to report write errors to user space. [...] If some process was copying a lot of data to that drive, the result will be an accumulation of dirty pages in memory, perhaps to the point that the system as a whole runs out of memory for anything else [...] a fair amount of attention was paid to the idea that write failures should result in the affected pages being kept in memory, in their dirty state. But the PostgreSQL developers had quickly moved on from that idea and were not asking for it".
> It is broken but there's no behavior that isn't, and the Postgres developers quickly understood why changing Linux's behavior isn't really possible.
I think there's still a fair number of PostgreSQL developers who think the way Linux kernel behaves makes the fsync() API rather difficult to use for anything but the simplest scenarios.
The reason why the community decided to accept it was the realization that there's about 0.001% chance of convincing kernel devs to change it, and the fact that we'd still have to deal with existing kernels in foreseeable future.
> Every computer component can fail in arbitrary ways, including drives. > > If you’re not robust against that, then when things like fsync fail, then you’ll lose availability and/or data.
The fact is that often the I/O issues are temporary, and those situations are becoming more and more common (think running out of disk space with thin provisioning, networking issues with NFS, etc.). So it might be quite valuable to handle those issues gracefully, without essentially crashing the database (which is pretty much what PANIC does).
> Even though Linux’s fsync behavior is clearly broken, it is far from the craziest behavior I’ve seen from the I/O stack.
"clearly broken" might be bit too harsh, but it certainly makes it way harder to use.
> Anyway, the main lesson here is that untested error handling is worse than no error handling. They should have figured out how to test that this path actually proceeds correctly (on real, intermittently failing hardware) or just panicked the process.
That's true, of course. It's a sad fact of life that error paths are the least tested part of almost any code base. It's however also true that testing I/O errors is pretty difficult to do (especially before "error" dm target), and a significant part of the fsyncgate was that kernel was not reporting errors reliably. It's also true that the behavior is somewhat filesystem-specific (some keep the data in page cache but marked as "not dirty", some will discard the data, ...). That makes testing pretty hard.