Comment by tedsanders
4 days ago
Interestingly, this is actually a question that's been looked at empirically!
Take a look at this paper: https://scholar.harvard.edu/files/rzeckhauser/files/value_of...
They took high-precision forecasts from a forecasting tournament and rounded them to coarser buckets (nearest 5%, nearest 10%, nearest 33%), to see if the precision was actually conveying any real information. What they found is that if you rounded the forecasts of expert forecasters, Brier scores got consistently worse, suggesting that expert forecast precision at the 5% level is still conveying useful, if noisy, information. They also found that less expert forecasters took less of a hit from rounding their forecasts, which makes sense.
It's a really interesting paper, and they recommend that foreign policy analysts try to increase precision rather than retreating to lumpy buckets like "likely" or "unlikely".
Based on this, it seems totally reasonable for a rationalist to make guesses with single digit precision, and I don't think it's really worth criticizing.
Likely vs. unlikely is rounding to 50%. Single digit is rounding to 1%. I don't think the parent was suggesting the former is better than the latter. Even before I read your comment I thought that 5% precision is useful but 1% precision is a silly turn-off, unless that 1% is near the 0% or 100% boundary.
The book Superforecasting documented that for their best forecasters, rounding off that last percent would reliably reduce Brier scores.
Whether rationalists who are publicly commenting actually achieve that level of reliability is an open question. But that humans can be reliable enough in the real world that the last percentage matters, has been demonstrated.
Your comment is incredibly confusing (possibly misleading) because of the key details you've omitted.
> The book Superforecasting documented that for their best forecasters, rounding off that last percent would reliably reduce Brier scores.
Rounding off that last percent... to what, exactly? Are you excluding the exceptions I mentioned (i.e. when you're already close to 0% or 100%?)
Nobody is arguing that 3% -> 4% is insignificant. The argument is over whether 16% -> 15% is significant.
3 replies →
The most useful frame here is looking at log odds. Going from 15% -> 16% means
-log_2(.15/(1-.15)) -> -log_2(.16/1-.16))
=
2.5 -> 2.39
So saying 16% instead of 15% implies an additional tenth of a bit of evidence in favor (alternatively, 16/15 ~= 1.07 ~= 2^.1).
I don't know if I can weigh in on whether humans should drop a tenth of a bit of evidence to make their conclusion seem less confident. In software (eg. spam detector), dropping that much information to make the conclusion more presentable would probably be a mistake.
I thought single digit means single significant digit, aka rounding to 10%?
I did mean 1%, not sure if I used the right term though, english not being my first language.
Wasn't 16% the example they were talking about? Isn't that two significant digits?
And 16% very much feels ridiculous to a reader when they could've just said 15%.
4 replies →
Aim small, miss small?