Comment by a-dub
7 years ago
If fsync() fails isn't valhalla lost anyhow? If you can't write things down because your pencil is broken, probably time to stop what you're doing and get a new pencil.
If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?
> If the kernel can't flush dirty write buffers, maybe it's time to send up a flag and panic in the kernel itself?
Being unable to write to a disk is a recoverable scenario, especially in some conditions. The most common cause of disk write failures is "someone yanked the flash drive out of the computer," and the recovery is "pop up a dialog telling the user to put it back in."
Another situation: a GCSFuse volume dealing with temporary networking errors.
Or NFS being NFS.
That would be a great improvement over the ‘tell the user he shouldn’t have done that’ we have now!
I think there is a case for never throwing away dirty pages.
Ie. either the write eventually succeeds, or all other writes will block or fail until it does.
That can happen when you remove a removable storage device, like a USB drive. You don't want to panic the kernel for that.
FreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious.
Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
Isn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open.
Redoing failed writes should be the method that ends in the least amount of corruption.
2 replies →
Reminds me. On CPM, hit save when the drive door is open, so sorry. All gone.
So kernel panic on disk full errors? Because that's one of the scenarios causing data loss here - fsync throwing away data on a full disk and lying about having committed it.
In that thread, Craig Ringer and Thomas Munro notes most systems & FS won't report ENOSPC on fsync(), they'll do that on write():
> ENOSPC doesn't seem to be a concern during normal operation of major file systems (ext3, ext4, btrfs, xfs) because they reserve space before returning from write(). But if a buffered write does manage to fail due to ENOSPC we'll definitely see the same problems. This makes ENOSPC on NFS a potentially data corrupting condition since NFS doesn't preallocate space before returning from write().
no, the fs doesn’t overcommit storage like the memory system does.
I don't think you understand the problem.
mmm, maybe this is a little too aggressive there could be perfectly functional storage on the host.
maybe something like "make the mount read only and ensure all open fds on the device return nothing but error and then dump the dirty blocks to swap in some kind of recovery log that could potentially be applied to the disk"
no, some errors are transient. a retry is a reasonable thing to be able to do in such a case. this happens with storage on SANs for example, due to a network blip.
if you've got removable disks on your servers, valhalla was lost long ago!
Huh? Hot swapping disks is extremely common.
Depends on your definition of 'hot' (and also on your definition of 'disk'). If you mean 'FS volume under active write load' then no it isn't extremely common.
the raid device remains stable and fsyncs don't fail in that case.
You don't use networked filesystems?
not on production db servers, no.