Comment by wruza
7 years ago
FreeBSD lived with that up to circa 2010, afair. Few fast-pulling desktop linux trolls complained at the time, but nothing serious.
Anyway, if given such choice, as a user/admin I would prefer an instant fail with a panic message, rather than a corrupted 10gb online client database with no clear recovery path. I’m always scared by this kind of story when something fails to fail underneath and I even didn’t ever hear of it.
Isn't your data already corrupted most of the time when writes start to fail? If the drive is gone, a panic won't help or hurt, it will just ruin everything else you have open.
Redoing failed writes should be the method that ends in the least amount of corruption.
Not going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed.
I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least.
“Redoing failed writes should be the method that ends in the least amount of corruption.”
But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.
> But in fact it led to a database corruption
You're saying you had a system that did redo failed writes to disk, unlike other systems discussed so far, and it managed to corrupt your database anyway?