Ah nice. I was dealing with that in one of our environments where a security update ended up causing apt to use more memory than usual so the oom killer nuked our elasticsearch process to "free up some memory".
And since that happened on all nodes around the same time, it took out the entire cluster. If you are not familiar, random, uncontrolled node restarts in any kind of multi node database or search product are a great way to trigger outages. So, not great.
I've had quite a few encounters with the oomkiller killing processes that were important and didn't need killing. Or as I like to phrase it "killing the one reason this server exists".
These days the way to size a server is "have enough memory to run whatever you need running + at least half a GB for whatever apt might randomly demand at any point". And guess what, memory tends to be expensive in cloud environments so people tend to get vms with as little as half a GB of ram.
Ah nice. I was dealing with that in one of our environments where a security update ended up causing apt to use more memory than usual so the oom killer nuked our elasticsearch process to "free up some memory".
And since that happened on all nodes around the same time, it took out the entire cluster. If you are not familiar, random, uncontrolled node restarts in any kind of multi node database or search product are a great way to trigger outages. So, not great.
I've had quite a few encounters with the oomkiller killing processes that were important and didn't need killing. Or as I like to phrase it "killing the one reason this server exists".
These days the way to size a server is "have enough memory to run whatever you need running + at least half a GB for whatever apt might randomly demand at any point". And guess what, memory tends to be expensive in cloud environments so people tend to get vms with as little as half a GB of ram.