Comment by kmeisthax

3 days ago

A read that happens to touch a particular torment nexus fd is still a long-running syscall, even if the syscall servicing routine itself is not long-running. The underlying problem is that program code that is "in a syscall" or "in an instruction" is in a special state for which interruption might not be possible or implemented well[0].

[0] Remember ITS and the PC2 problem?

6 comments

kmeisthax

Reply
EvanAnderson  3 days ago

For other people who didn't immediately recognize "ITS and the PC2 problem" and want the backstory it starts in the last paragraph of page 312: https://web.mit.edu/~simsong/www/ugh.pdf

  • inigyou  2 days ago

    The explanation is pretty bad. I believe this is about Unix signals. If a signal occurs, the kernel will push a signal stack frame on the user-mode stack and send the CPU to user mode to run the signal handler. When it returns it returns to the point the signal occurred. There is probably some glue to restore register values.

    But if a signal comes inside a syscall the user-mode program counter is the syscall instruction, not the exact position in kernel mode within the syscall. What should the kernel push on the stack? Obviously it can't push the kernel PC as that would be a huge vulnerability, and it would lose all the state on the kernel stack anyway. If the syscall is a quick one like getpid, it can just finish the syscall and then do the signal, but if it's read, then it's a problem.

    The proper solution is for read to somehow save its state, store the user PC of the syscall instruction, then exit the syscall and do the signal, and when the signal is done it goes back to the syscall. This is doable enough for read, since you just advance the buffer and decrease the length, though you still need a way to return the correct total number of bytes. It's completely infeasible for anything more complicated than that, like many ioctls.

    So instead the worse-is-better solution was used. If read gets a signal, it turns itself into a "quick" syscall by just giving up on waiting for more bytes and returning whatever it has already read, which may be 0 bytes. It finishes immediately, does the syscall and returns to the syscall's caller. It is the application's problem to deal with the fact this can happen.

    On Windows NT they can actually mix kernel and user stack frames arbitrarily. User code can call into kernel code that can call into user code that can call into kernel code, etc, and kernel debuggers can see the whole thing. I have no idea how they do this. Unix doesn't - Unix is strictly user code calling into kernel code via syscalls.

touisteur  3 days ago

I have a reproducible way to have a pwrite syscall on a specific SSD on a specific machine take 15+ seconds and completely block any syscall related to that SSD by any other thread or core during that amount of time. I tried and couldn't preempt it either (sched_fifo and preempt kernel options). I should have a look soon with Intel PT to check whether it's on the same instruction every time :)

  • inigyou  2 days ago

    Won't be the same instruction. It'll be waiting for a hardware interrupt without properly preparing to handle signals at the same time because someone assumed it was fast. Kernel code isn't held together with any less duct tape than user code, although we hope it crashes less.

    In Linux, any thread running in the kernel is unkillable unless that section of kernel code made arrangements to be killable. When kernel code blocks, you can get unkillable processes. They show as D state (uninterruptible wait).