Comment by giantrobot
3 years ago
> I don’t recall how this worked
Unlinking files reduces their reference count. Once the reference count reaches zero the file is considered deleted and the space can be reclaimed. Every open file descriptor and hard link increases a file's reference count. So if you've got a file descriptor open on a file in a background task (daemon, nohup, screen, etc) and unlink it with rm or something the file's reference count will decrement but not go to zero. Only when that program closes the file descriptor will the reference count go to zero and the file actually be deleted.
I had an interesting issue with that the other day:
Someone on my company thought it good to launch background process from within a cron, and start it with `&ˋ at the end, so that it detached from the cron and goes as child to init .
Right, but the program opens std out and dumps its output in it. Guess what, the fd handling the std out ends up in a temp file which is deleted by cron, but still held by the program.
I ended up with a full / in all production servers, with no culprit when running ˋdu`. Only running ˋlsof |grep deleted` did I find these huge temporary files.
Killing the process and switching to a saner systemd service was the savior.