Comment by throw0101c
2 days ago
> The article speculates on why Apple integrates the SSD controller onto the SOC for their A and M series chips, but misses one big reason, data integrity.
If they're really interested with data integrity they should add checksums to APFS.
If you don't have RAID you can't rebuild corrupted data, but at least you know there's a problem and perhaps restore from Time Machine.
For metadata, you may have multiple copies, so can use a known-good one (this is how ZFS works: some things have multiple copies 'inherently' because they're so important).
Edit:
> Apple File System uses checksums to ensure data integrity for metadata but not for the actual user data, relying instead on error-correcting code (ECC) mechanisms in the storage hardware.[18]
* https://en.wikipedia.org/wiki/Apple_File_System#Data_integri...
> If they're really interested with data integrity they should add checksums to APFS.
Or you can spend half a billion dollars to solve the issue in hardware.
As one of the creators of ZFS wrote when APFS was announced:
> Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices. Both NAND flash SSDs and magnetic media HDDs use redundant data to detect and correct errors. The Apple engineers contend that Apple devices basically don't return bogus data.
https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana...
APFS keeps redundant copies and checksums for metadata, but doesn't constantly checksum files looking for changes any more than NTFS does.
> Or you can spend half a billion dollars to solve the issue in hardware.
And hope that your hardware/firmware doesn't ever get bugs.
Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth.
ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down":
* https://www.youtube.com/watch?v=fE2KDzZaxvE
And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all.
When the vast majority of the devices you sell run on battery power, it makes far more sense from a battery life perspective to handle issues in hardware as much as possible.
For instance, try to find a processor aimed at mobile devices that doesn't handle video decoding in dedicated hardware instead of running it on a CPU core.
3 replies →
That solution doesn't help anyone who's using external storage, though, so it kinda feels like a half billion dollars spent on a limited solution.
There is nothing preventing you from running OpenZFS on external storage if you are worried that the hardware you purchased is less reliable.
7 replies →
Apple does not care about external storage at all, as in external disks. They offer iCloud for external storage. They don't sell external disks. They don't like cables. They make lots of money selling you a bigger internal disk.
1 reply →
No one requires you to use APFS for your external storage!
4 replies →
maybe apple doesn't want you to use external storage, because storage size is how apple upsells devices and grabs larger premium.
By using external storage, instead of paying $10k more for more storage, you are directly harming Apple’s margins and the CEO’s bonus which is not ok /s
1 reply →
That is a weak excuse to rely on data integrity in the hardware. They most likely had that feature and removed it so they wouldn't be liable for a class action lawsuit when it turns out the NAND ages out due to bug in the retention algorithm. NTFS is what, 35 years old at this point? Odd comparison.
The point is that NTFS makes zero effort to maintain file integrity at any level.
Handling file integrity at the hardware level is a big step up.
1 reply →
Believing that giant companies are monolithic “theys” leads to all sorts of fallacies.
Odds are very good that totally different people work on the architecture of AFS and SoC design.
Even still, those people report to people that report to people until you eventually get to the person in charge of the full product.
You can do this yourself in userspace if you really want it:
https://git.eeqj.de/sneak/attrsum
I use zfs where I can (it has content checksums) but it sucks bad on macOS, so I wrote attrsum. It keeps the file content checksum in an xattr (which APFS (and ext3/4) supports).
I use it to protect my photo library on a huge external SSD formatted with APFS (encrypted, natch) because I need to mount it on a mac laptop for Lightroom.
A similar alternative is Howard Oakley’s Dintch/Fintch/cintch:
https://eclecticlight.co/dintch/
Worth noting, for ZFS - you can use the "copies" property of the dataset to save 2 or (usually) 3 separate copies of your data to the drive(s).