Comment by rancar2

TLDR: I was called in several hours after the downtime started. I architected and built the full system so I knew things were incredibly consistent and how every layer worked (each layer of the software/database, load balancers, the quirky legacy protocols, network setup, etc). IBM was saying it was an issue on the financial institution side, which I validated was not the case practically first (things were down so I was given freedom by the executive team to do as I needed). Since I sanity checked the easy parts first and had high confidence this was something much more wild after scanning through more stack traces than a human should, I decompiled and reverse engineered their code to show the IBM teams where the bug was in their switch statements triggered by the Redhat patch difference and its dependencies that triggered the edge case that stopped the highly available cluster setup from functioning. I created a script to override the system properties being read in on startup to trick the switch statement, show it worked, and got the system back online. IBM made an official hot patch that fixed the issue officially along with paying a lot of money back to the financial institution for their troubles (ie the double contractual obligations for the premium IBM support with included SLA clauses that were overrun and the proprietary IBM software itself).