Comment by Bender

I've seen something like this before. A monitoring team said that front-line engineers should be opening tickets for every alert that came in. There were upwards of 10,000 unique alerts per hour. Critical issues had been missed due to all the noise. The directors and VPs got an earful enough times that they forced the monitoring team to open tickets for all the alerts and troubleshoot them. The next day the system was down to about a dozen unique alerts per hour after excluding informational alerts and the remainder were eventually closed out. It was the first time the boards had all been green for real this time vs. painting them green for the media and the Gartner Group.