Comment by ungreased0675
9 hours ago
Fascinating project. How are the probabilities being measured? I’m skeptical of AI analysis because my past experience is that it appears to be analyzing, but behind the curtain there’s not even a fraction of the same rigor a human analyst would have done. In other words It’s very shallow but looks deep.
In this model the probabilities were measured by an ensemble of LLM's. We give them context and ask them to follow a specific methodology for making a forecast, and then aggregate their forecasts into a "crowd" forecast. We can get input from humans as well to compare/combine them further, but in this particular instance I just used our AI Forecaster.
You're right that a human analyst - especially a very experienced one - is going to be able to still do a better job of analysis most of the time, but the AI is useful for having loops to re-examine the question, consume and curate new information, etc. We've built the system with that assumption: let humans do what they're best at, let AI do what it's best at...