Comment by yakhinvadim

I actually agree that "true" significance can only be estimated in hindsight.

My goal with this project is not to get "true" significance but to have a setup that gets you 90% of the way there: an automated system that finds events that are likely to affect large groups of people or major systems and filters out most of everyday noise.

There will always be false-positives and false-negatives, but I think it's a good starting point and it should slowly get better as models get smarter.