Comment by whatever1
2 days ago
I dont know how one can objectively evaluate forecasting. We observe just one universe. We overfit our models to this one observation we have.
Which is why at the end of the day why nobody trusts or blames the forecasting models, specially in all or nothing types of decisions.
You can't easily evaluate the success of any single forecast, but as soon as you have a collection of forecasts there are a lot of ways to evaluate which forecasters are more accurate and which are well calibrated in their predictions.
For example you can look at the calibration of the forecasts: Take all the outcomes where you estimated ~10% probability. How many of them actually happened? If you are an well-calibrated forecaster, the actual number should be fairly close to 10%.
Brier Score:
This is the pseudo-intellectual version of your boomer relative ranting about the weatherman on TV, “how does this guy still have a job when he’s always wrong!”
Or the poll-bashing that happened after the 2016 election when an outcome with a 29% chance per one model was observed.
Any interesting forecasting problem is underconstrained. Monte Carlo methods are the classic way to mitigate that one-universe problem.
Nobody can predict the future with certainty. All models are wrong, but some are useful.
How does Monte Carlo help mitigation?
I fully agree that all models are wrong.
Their main usage is just replaying the past or encoding the modelers assumptions for the future.
But there is no way to know if a model is good or bad.
There would be if we had infinite universes.
Monte Carlo methods are used to estimate the risk associated with a prediction (and presumably an action taken against that prediction). Does that answer your question? I can elaborate if you like.
2 replies →