Comment by hinkley

It's also worth writing yourself some smoke tests that tell you when your very important customer starts sending total garbage to your servers and expects you to just handle it because they're paying you instead of the other way around. Even a couple hours of additional warning that you're about to get pulled into an uncomfortable conference call can greatly improve outcomes.