Comment by Twirrim
7 hours ago
NUMA is one of those amazing things that trip you up in all sorts of ways at unexpected times. The amazing "invisible" performance killler (invisible because unless you're already aware of NUMA, or remember to check, you won't know it's there potentially crippling you.)
It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too late.
I don't know if the kernel has improved this behaviour in the several years since last tested, but a coworker realised that the linux page-cache wasn't fully split by NUMA node. They were benchmarking mysql running it in each NUMA node, and noticed the second NUMA node was noticeably slower. Then discover after a reboot the second node was fast, and the first was slower. After a bit of thinking and tinkering they discovered that libmysql was ending up in the page cache in the same NUMA node as the benchmark client was run in first, so even though they were pinning the benchmark tool and mysql process to the NUMA node, the benchmark client was causing the OS to reach across the NUMA node to get at the page cached library.
It's unfortunately a hard message to sell, but your program itself should never be backed by a file, for this exact reason. All code should be remapped into anonymous memory that is 1) right where you want it; 2) backed by hugepages; and 3) not shared. You are gaining nothing by trying to share objects with other processes, and you are losing a great deal of performance. The tradeoff made a little sense in the 1980s when they came up with it but in this century it simply doesn't.