Comment by kqr
4 days ago
It is producing a useful number. As more truly independent terms are added, error grows with the square root while the point estimation grows linearly. In the aggregate, the error makes up less of the point estimation.
This is the reason Fermi estimation works. You can test people on it, and almost universally they get more accurate with this method.
If you got less certain of the result in the example, that's probably a good thing. People are default overconfident with their estimated error bars.
Read a bit on Fermi estimation, I'm not quite sure exactly what the "method" is in contrast to a less accurate method, it's basically just getting people to think in terms of dimensional analysis? This passage from the Wikipedia is interesting:
By contrast, precise calculations can be extremely complex but with the expectation that the answer they produce is correct. The far larger number of factors and operations involved can obscure a very significant error, either in mathematical process or in the assumptions the equation is based on, but the result may still be assumed to be right because it has been derived from a precise formula that is expected to yield good results.
So the strength of it is in keeping it simple and not trying to get too fancy, with the understanding that it's just a ballpark/sanity check. I still feel like the Drake equation in particular has too many terms for which we don't have enough sample data to produce a reasonable guess. But I think this is generally understood and it's seen as more of a thought experiment.
> People are default overconfident with their estimated error bars.
You say this but yet roughly in a top level comment mentions people keep their error bars too close.
Sorry, my comment was phrased confusingly.
Being overconfident with error bars means placing them too close to the point estimation, i.e. the error bars are too narrow.
Ah right thanks, I read that backwards.
They are meaning the same thing. The original comment pointed out that people’s qualitative description and mental model of the 95% interval means they are overconfident… they think 95 means ‘pretty sure I’m right’ rather than ‘it would be surprising to be wrong’