Comment by diseasedyak
18 hours ago
Yah, prediction interval instead, right? (I'm still learning statistics)
Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
If 95% of the intervals in your set of intervals include μ, and you randomly pick one of them, in what way is that interval not 95% likely to contain μ? Ignoring the frequentist pedantry that "likelyhood is the wrong word", is there a way for a different number to be the correct number?
https://link.springer.com/article/10.3758/s13423-015-0947-8#...
https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... (+ Cauchy example below)
It’s like with test accuracy. Test accuracy is the pre-test probability that the test will give a correct result. But once you have a positive or negative result, which way it turned out plays a part in computing the predictive value. Likewise, once you have computed the interval, the specific bounds you ended up getting can affect the plausibility that they contain the true value.
4 replies →
> a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well
Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.
Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.
Does that mean it's valid? Well, the confidence interval is as equally valid as a lot of credible intervals, which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it, it is not a different procedure that produces different numbers.
> Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.
The confidence interval is “valid” for what its definition is. But:
- saying “there is a 95% chance that this interval contains the true value because it’s a 95% credible interval” is valid;
- saying “there is a 95% chance that this interval contains the true value because it’s a 95% confidence interval” is not.
It’s very much possible to reach a correct conclusion with invalid reasoning. As Jaynes put it:
“As we stress repeatedly, from a false proposition – or from a fallacious argument that leads to a false proposition – all propositions, true and false, may be deduced. But this is just the danger; if fallacious reasoning always led to absurd conclusions, it would be found out at once and corrected. But once an easy, shortcut mode of reasoning has led to a few correct results, almost everybody accepts it; those who try to warn against it are not listened to.”
> Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.
Right, but if you get that by happenstance and not from the fact that it’s a confidence interval, the latter isn’t really getting you anything.
Per my first link: “One of the misconceptions regarding the relationship between Bayesian inference and frequentist inference is that they will lead to the same inferences, and hence all confidence intervals can simply be interpreted in a Bayesian way. In the case where data are normally distributed, for instance, there is a particular prior that will lead to a confidence interval that is numerically identical to Bayesian credible intervals computed using the Bayesian posterior (Jeffreys 1961; Lindley 1965). This might lead one to suspect that it does not matter whether one uses confidence procedures or Bayesian procedures. We showed, however, that confidence intervals and credible intervals can disagree markedly. The only way to know that a confidence interval is numerically identical to some credible interval is to prove it. The correspondence cannot — and should not — be assumed. More broadly, the defense of confidence procedures by noting that, in some restricted cases, they numerically correspond to Bayesian procedures is actually no defense at all. One must first choose which confidence procedure, of many, to use; if one is committed to the procedure that allows a Bayesian interpretation, then one’s time is much better spent simply applying Bayesian theory. If the benefits of Bayesian theory are desired — and they clearly are, by proponents of confidence intervals — then there is no reason why Bayesian inference should not be applied in its full generality, rather than using the occasional correspondence with credible intervals as a hand-waving defense of confidence intervals.”
> which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it
You don’t get to choose to interpret a confidence interval as if it had the meaning of a Bayesian one. If you want to be able to interpret a computed interval as “there is a 95% chance that this interval contains the true parameter value”, only the Bayesian procedure guarantees it, and it can very well produce different numbers. So it is a better alternative in that sense.
[flagged]