Comment by happyPersonR

Like 3/4 of my career was building stuff like this along with the software that ran in it for basically this exact use case. We even had it integrated into our CI. Spent upwards of a decade building things like this.

One thing we realized was that we needed to be able to simulate upgrades like this.

Lead to a lot of complexity but it was worth it.

This testing was some of the most valuable and because we had it automated it really let us build better things faster and find issues and regressions and verify new functionality :)