uncertainty-quantification

uncertainty-quantification

When to invoke

• Parsing statistical claims from research papers (σ, p-value, Bayes factor, credible interval, confidence interval)
• Writing your own outputs — deciding between "certain", "likely", "possible"
• Composing probabilities from multiple independent signals
• Auditing an LLM classifier that outputs binary yes/no when a probability would be more useful
• Reconciling frequentist and Bayesian claims ("this is 3σ" vs "the posterior concentrates at…")

The 5 families of uncertainty you'll encounter

1. Statistical (aleatoric) — measurement noise, sampling variance. Quantified by error bars, σ, confidence intervals.
2. Systematic — instrumental bias, modeling approximations, calibration drift. Usually larger than stat and harder to quote.
3. Epistemic — lack of knowledge that could be reduced with more data or better models. What Bayesian priors encode.
4. Model uncertainty — is the underlying model even right? (Often the largest uncertainty, almost never quoted)
5. Linguistic / representational — "we observed" vs "we infer" vs "we estimate". Words that look precise but hide choices.

Decoding "3σ" (frequentist)

• σ = standard deviation. "3σ detection" = ~99.73% of Gaussian bulk; p ≈ 2.7×10⁻³ one-tailed, 2.7×10⁻⁴ two-tailed.
• Local vs global significance matters enormously:
  • Local: p-value at one specific look
  • Global: corrected for the "look-elsewhere effect" across many trials
  • A paper claiming "3σ local, 1.5σ global" = essentially not a detection.
• HEP convention:
  • 2–3σ: "evidence for"
  • 3–5σ: "strong evidence" (but still one trial-factor away from noise)
  • ≥5σ: "observation" / "discovery"
• Cosmology/astro convention is more relaxed; 3σ often counted as a result.

Red flag: a claim of "3σ" without stating local-vs-global is almost always local only.

Bayesian alternatives

• Credible interval (CI): "with probability 95%, θ ∈ [a, b] given the data and prior"
• Bayes factor (BF): ratio of likelihoods under competing hypotheses
  • BF < 3: barely worth mentioning
  • 3 ≤ BF < 20: moderate evidence
  • 20 ≤ BF < 150: strong evidence
  • BF ≥ 150: decisive
• Posterior probability: P(hypothesis | data) directly — the quantity most people think σ represents but doesn't.