← Back to context

Comment by marcan_42

4 years ago

    #!/usr/bin/python
    import os, sys, time, datetime

    t = datetime.datetime.now().isoformat()

    print(t)
    for i in range(5):
        time.sleep(1)
        print(4 - i)

    fd = os.open(sys.argv[1], os.O_RDWR|os.O_CREAT)
    os.lseek(fd, 0, 0)
    os.write(fd, b"test: " + t.encode("ascii") + b"\n");
    os.fsync(fd)
    print("done!")
    time.sleep(100)

Run that on a Mac Mini. Do it a couple times. Remember the timestamp of the last one. Let it count down, then pull the plug within a few seconds after "done!" shows up. Boot up again. The file contents will have reverted to a prior point in time.

This isn't some hypothetical thing, this is a trivial test you can do. fsync() on macOS does not guarantee data is on stable storage. And this is actually well documented.

Then if you want to see the performance problem, make it a loop instead and use `fcntl.fcntl(fd, fcntl.F_FULLFSYNC, 1)`. You'll get 40-odd IOPS, but at least your data won't disappear after power loss.

Does it somehow prove that people are losing files when transferring them over the network? People would definitely notice it.

  • You're very persistent, aren't you?

      marcan@raider:~/tmp -$ echo "very important data" > file.txt
      marcan@raider:~/tmp -$ rsync --remove-source-files file.txt macmini.lan:
    

    Yanked power to macmini.lan after the rsync completed, then turned it on again

      marcan@raider:~/tmp -$ ls file.txt
      ls: cannot access 'file.txt': No such file or directory
      marcan@raider:~/tmp 2$ ssh macmini.lan
      Last login: Thu Feb 17 23:45:26 2022 from 192.168.3.10
      marcan@Mini-M1-2020 ~ % ls file.txt
      ls: file.txt: No such file or directory
    

    Data's gone.

    This is real, please stop pretending it isn't.

    • At least I’m not as rude as you. You’ll need to be persistent to accomplish your results with the Asahi Linux project - good luck with that.

      I had no doubts that you can lose your file if you are moving it and some very lucky power outage hits. I was not “pretending” it's not real.

      What I have doubts about, is that it's a real concern for 99.93% of the users. As we’ve found here, it's not even a rare case in other kinds of OS, so users would definitely notice it.

      It is theoretically possible, of course. In practice, it's just too rare to consider.

      But still, I hope this topic will be noticed by Apple and they fix that low performance issue for fullsync. Also, I hope they will not make fullsync as the default behavior - it doesn't worth the risks (and those for whom it does - should use some Linux for sure).

      5 replies →