← Back to context

Comment by wruza

7 years ago

Not going into details, at the end of the day one should have a plan to recover their value from a potential failure. By the means of version control systems, transaction logs, backups, mirrors, etc. But if a failure makes its way to a recovery storage undetected, the entire plan is doomed.

I don’t care if I lose few last hours of my local work, since this time is buffered in a project or is not precious at all (like game saves). That throws back me a little, but not anyone else. I also don’t really care about losing downloads or experimental files. But when it comes to a business database or a project repository, that value can be orders of magnitude bigger and can cost responsible one their job at the very least.

“Redoing failed writes should be the method that ends in the least amount of corruption.”

But in fact it led to a database corruption that was only fixed by some black magic. As far as I understood, they did manual rollback of failed writes. I wouldn’t blame anyone for subj bug even if met it by myself — writing low-level software is hard, the behavior was unintended, and both linux, xfs & pg are free as in beer and free of any charges. But if it happened and was not detected for longer than a recovery window allows it, costs were imminent. That’s why this trade-off is easy on a distance.

> But in fact it led to a database corruption

You're saying you had a system that did redo failed writes to disk, unlike other systems discussed so far, and it managed to corrupt your database anyway?