Comment by seiji
13 years ago
That's why the D in ACID confused me for a long time. There's no way to guarantee durability. They say "durable" but they mean "durable, kinda, we think, we hope, if nothing is lying to us, if your storage array doesn't lose power without a battery to flush the cache, if your system doesn't power down while your drives are lying about their committed write status, or if your write succeeded then an elf ran away with your platters."
In days of old from ACID was dreamt of, computers were built head to tail by giant engineering firms largely with three letter names. In those days, when you wrote to disk, it was on disk. If you handed it to a subsystem to write to disk, you could believe the answer because your people built that subsystem and it had better do what you wanted.
Now, computers are built with subsystems from a half dozen different companies, none of which can be bothered to fully document their product's behavior. Product life cycles are so fast it isn't worth working out what a device actually does, because by the time you do you either won't be able to buy it anymore, or someone's new iteration will be 20% cheaper/faster/bigger and you will get killed in the market if you don't switch to it.
␄
PS: We were also happy to have a whole MIP of Vax equivalent (or identical) processing power, and I was slightly reknowned for my ability to code cleverly enough to get 42 disk IO/second out of our main disk drive, so I'd rather not go back, think you very much. I'll get along as best I can with the miraculous and inexpensive rubbish we build our systems from today.
> when you wrote to disk, it was on disk.
So, if the drives didn't lie about a flush that problem is solved. But if it's on disk at time t, that's no guarantee that you can read it back at time t+1. The drives can physically fail. These "days of old" are before my time, but I really doubt they had magic disks that never physically degraded.
The D is "durable", not "will survive the apocalypse along with the cockroaches". I've got plenty of things I'd describe as durable while being capable of breaking.
1 reply →
Well it means the system tries the best it can do make the value durable. It will do a sync after every write perhaps. If the disk lies, well there is not much sqlite can do. If disk has a capacitor and battery maybe it can still lie that it wrote but keeps it in its internal buffer and if someone pulls the cord it has enough energy to flush everything it "lied about" to disk. Oh but what if you shoot a proton beam through it and blow away its capacitor and battery, well then you are out of luck.
As someone mentioned, this is contrast to say how MongoDB was shipping not too long ago. They had turned of any acknowledgement for writes. So doing a db write was more like a throw over the fence and pray operation. So it didn't even try to be durable, by design.
Sqlite guarentees Durability by waiting and checking the disk to make sure it is stored correctly before reporting a success. This default behavior is rather slow and can be turned off.
This is in stark contrast to something like MongoDB, which barely parses the request before reporting a success, and doesn't make any guarantees of when or even if it will ever save the data (though it usually does).
What happens if the disk or kernel lie. At both levels, I can see a potential performance enhancement by buffering changing while transparently acting as if they have actually been committed. My guess would be that hard-drives have explicit instruction(s) than ask for what actually happened; but I have never worked with this type of thing.
> What happens if the disk or kernel lie
My understanding is that the kernel (for Windows and Linux values of "kernel") will never lie to you. They will accurately report what the storage drivers told it, and the lying occurs at that level.
Storage drivers seem to be a big source of corruption for MSSQL [1]:
> The most common cause of database corruption (more than 95% of all corruption cases) that we in PSS encounter turn out to be caused by a platform issue, which is a layer below the SQL Server. The most common individual cause is a 3rd party driver or firmware bug.
And as well
> My guess would be that hard-drives have explicit instruction(s)
They do, but the problem is that the storage drivers lie about what actually happened. E.g. Basically they implement write caching for the "flush the write cache" instruction.
[1] http://blogs.msdn.com/b/suhde/archive/2009/04/08/introductio...
3 replies →