Comment by newnewpdro
7 years ago
> And people realized the Linux I/O freeze is a variant of bufferbloat, and the very same ideas of the CoDel algorithm can be applied to the Linux I/O freeze problem.
There are myriad causes for poor interactivity on Linux systems under heavy disk IO. I've already described the two I personally observe the most often in another post here [1], and they have nothing at all in common with bufferbloat.
Linux doesn't need to do less buffering. It needs to be less willing to evict recently used buffers even under pressure, more willing to let processes OOM, and a bridging of the CPU and IO scheduling domains so arbitrary processes can't hog CPU resources via plain IO on what are effectively CPU-backed IO layers like dmcrypt.
But it gets complicated very quickly, there are reasons why this isn't fixed already.
One obvious problem is the asynchronous, transparent nature of the page cache. Behind the scenes pages are faulted in and out on demand as needed, this generates potentially large amounts of IO. If you need to charge the cost of this IO to processes for informing scheduling decisions, which process pays the bill? The process you're trying to fault in or the process that was responsible for the pressure behind the eviction you're undoing? How does this kind of complexity relate to bufferbloat?
> It needs to be less willing to evict recently used buffers even under pressure, more willing to let processes OOM
I've had similar experiences. On Windows, where either through bugs or poor coding, an application requests way too much memory, leading to an unresponsive system while the kernel is busy paging away.
On Linux I've experienced the system killing system processes when under memory pressure, leading to crashes or an unusable system.
I don't understand why the OS would allow a program to allocate more than available physical memory, at least without asking the user, given the severe consequences.
Overcommit is a very deliberate feature, but its time may have passed. Keep in mind this is all from a time when RAM was so expensive swapping to spinning disks was a requirement just to run programs taking advantage of a 32-bit address space.
You can tune the overcommit ratio on Linux, but if memory serves (no pun intended) the last time I played with eliminating overcommit, a bunch of programs that liked to allocate big virtual address spaces ceased functioning.
Yeah, I know it was a feature at one point... but at least the OS should punish the program overcommitting, rather than bringing the rest of the system down (either by effectively grinding to a halt or killing important processes).