Comment by spider-mario
19 hours ago
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.
Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).
Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".
It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.
Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.
That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).
I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.
The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless.
I'm not that well versed in statistics, but a standard box plot is probably the best alternative
A single result is binary. All we get from a run is which tasks were solved, which weren’t.
Both ways involve sophistry. If you don't like dirty tricks, statistics isn't for you.
> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value).
They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.
It’s not reaching. See my response here: https://news.ycombinator.com/item?id=49622516
> Like saying the probability your uncle died of a shark attack is either 0 or 1.
For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t just stop at “it was drawn from a pool of intervals of which 95% contain the true value” since you have more information than that (the numerical bounds).
Task success is not enough; reliability and cost matter.
Yah, prediction interval instead, right? (I'm still learning statistics)
Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
If 95% of the intervals in your set of intervals include μ, and you randomly pick one of them, in what way is that interval not 95% likely to contain μ? Ignoring the frequentist pedantry that "likelyhood is the wrong word", is there a way for a different number to be the correct number?
4 replies →
> a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well
Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.
Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.
Does that mean it's valid? Well, the confidence interval is as equally valid as a lot of credible intervals, which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it, it is not a different procedure that produces different numbers.
1 reply →
[flagged]
Mind blown. The more I read about statistics, the less I know.
“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)
Thank you for these, coz I learned a lot! Great that they are open access.
the 2nd derivate of shannon covariance of noise begs to differ