ssbc.metrics.loo_uncertainty
Small-sample uncertainty quantification for LOO-CV conformal prediction.
This module handles all four sources of uncertainty: 1. LOO-CV correlation structure 2. Threshold calibration uncertainty 3. Parameter estimation uncertainty 4. Test sampling uncertainty
Designed for small calibration sets (n=20-40) where bootstrap is unreliable.
Functions
METHOD 1: Analytical bounds with LOO correction. |
|
METHOD 2: Exact binomial with effective sample size (CONSERVATIVE). |
|
METHOD 3: Distribution-free Hoeffding bound (ULTRA-CONSERVATIVE). |
|
Compute prediction bounds using Clopper-Pearson + sampling uncertainty with LOO correction. |
|
|
Main function: Compute robust prediction bounds for small-sample LOO-CV. |
|
Estimate the actual variance inflation from LOO-CV for this specific problem. |
|
Generate a formatted text report of prediction bounds. |
- ssbc.metrics.loo_uncertainty.estimate_loo_inflation_factor(loo_predictions, verbose=True)[source]
Estimate the actual variance inflation from LOO-CV for this specific problem.
Theory predicts inflation ≈ 2×, but the actual value depends on the model and data structure. This computes an empirical estimate.
Parameters:
- loo_predictionsnp.ndarray, shape (n_cal,)
Binary LOO predictions (1=success, 0=failure)
Returns:
- inflation_factorfloat
Estimated variance inflation, clipped to [1.0, 6.0] Typically ≈ 2.0 for LOO-CV
Notes:
Inflation = Var_empirical / Var_IID For n → ∞: inflation → 2.0 For small n: inflation can vary, so we clip to reasonable range
- ssbc.metrics.loo_uncertainty.compute_loo_corrected_bounds_analytical(loo_predictions, n_test, alpha=0.05, use_t_distribution=True, inflation_factor=None)[source]
METHOD 1: Analytical bounds with LOO correction.
This method: - Uses empirical variance of LOO predictions (captures correlation) - Applies theoretical LOO inflation as safety check - Uses t-distribution for small-sample critical values - Combines calibration and test sampling uncertainty
Best for: n_cal, n_test ≥ 40 AND inflation_factor ≥ 1.2
Note: When inflation_factor ≈ 1 (low correlation), the normal approximation becomes unreliable. Use Method 2 (exact binomial) instead in these cases.
Parameters:
- loo_predictionsnp.ndarray, shape (n_cal,)
Binary LOO predictions
- n_testint
Size of future test sets
- alphafloat
Significance level (e.g., 0.05 for 95% confidence)
- use_t_distributionbool
If True, use t-distribution (recommended for n < 50)
- inflation_factorfloat or None
Manual override for LOO inflation. If None, auto-estimated.
Returns:
- L_primefloat
Lower prediction bound
- U_primefloat
Upper prediction bound
- diagnosticsdict
Detailed breakdown of variance components
- ssbc.metrics.loo_uncertainty.compute_loo_corrected_bounds_exact_binomial(k_loo, n_cal, n_test, alpha=0.05, inflation_factor=2.0)[source]
METHOD 2: Exact binomial with effective sample size (CONSERVATIVE).
This method: - Uses exact beta/binomial distributions (no normal approximation) - Computes effective sample size accounting for LOO correlation - Uses worst-case union bound for combining uncertainties - Works directly with discrete probabilities
Best for: n_cal = 20-40, when you need guaranteed coverage
Parameters:
- k_looint
Number of LOO successes
- n_calint
Calibration set size
- n_testint
Test set size
- alphafloat
Significance level
- inflation_factorfloat
LOO variance inflation (typically 1.5-2.5)
Returns:
- L_prime, U_primefloat
Prediction bounds
- diagnosticsdict
Detailed breakdown
- ssbc.metrics.loo_uncertainty.compute_loo_corrected_bounds_hoeffding(loo_predictions, n_test, alpha=0.05, inflation_factor=None, verbose=True)[source]
METHOD 3: Distribution-free Hoeffding bound (ULTRA-CONSERVATIVE).
This method: - Uses Hoeffding concentration inequality (no distributional assumptions) - Accounts for LOO correlation via adaptive effective sample size - Provides guaranteed coverage regardless of distribution - Widest bounds, suitable as worst-case / sanity check
Best for: When you absolutely need guaranteed coverage
Parameters:
- loo_predictionsnp.ndarray
Binary LOO predictions
- n_testint
Test set size
- alphafloat
Significance level
- inflation_factorfloat, optional
LOO correlation inflation factor. If None, uses conservative default of 2.0.
Returns:
- L_prime, U_primefloat
Prediction bounds
diagnostics : dict
- ssbc.metrics.loo_uncertainty.compute_loo_corrected_prediction_bounds(loo_predictions, n_test, alpha=0.05, method='simple', inflation_factor=None, verbose=True)[source]
Compute prediction bounds using Clopper-Pearson + sampling uncertainty with LOO correction.
This function provides the correct statistical approach: 1. Use Clopper-Pearson to bound the true rate p from calibration data 2. Add sampling uncertainty for future test sets of size n_test 3. Account for LOO-CV correlation structure
- Parameters:
loo_predictions (np.ndarray, shape (n_cal,)) – Binary LOO predictions (1=success, 0=failure)
n_test (int) – Expected size of future test sets
alpha (float) – Significance level (e.g., 0.05 for 95% confidence)
method (str) – ‘simple’ - Clopper-Pearson + normal approximation for sampling uncertainty ‘beta_binomial’ - Exact Beta-Binomial approach ‘hoeffding’ - Distribution-free Hoeffding bounds
inflation_factor (float, optional) – Manual override for LOO variance inflation factor
verbose (bool, default=True) – Print diagnostic information
- Returns:
(lower_bound, upper_bound, diagnostics)
- Return type:
- ssbc.metrics.loo_uncertainty.compute_robust_prediction_bounds(loo_predictions, n_test, alpha=0.05, method='auto', inflation_factor=None, verbose=True)[source]
Main function: Compute robust prediction bounds for small-sample LOO-CV.
This is the primary entry point. It intelligently selects methods based on sample size and provides comprehensive diagnostics.
Parameters:
- loo_predictionsnp.ndarray, shape (n_cal,)
Binary LOO predictions (1=singleton/success, 0=not/failure)
- n_testint
Expected size of future test sets
- alphafloat
Significance level (e.g., 0.05 for 95% confidence)
- methodstr
‘auto’ - Automatically select best method (recommended) ‘analytical’ - Method 1: Analytical with LOO correction ‘exact’ - Method 2: Exact binomial with effective n ‘hoeffding’ - Method 3: Distribution-free bound ‘all’ - Compute all three and report
- inflation_factorfloat, optional
Manual override for LOO variance inflation factor. If None, automatically estimated. Typical values: 1.0 (no inflation), 2.0 (standard LOO), 1.5-2.5 (empirical range)
- verbosebool, default=True
If True, print diagnostic information about method selection and inflation factors.
Returns:
- L_primefloat
Lower prediction bound
- U_primefloat
Upper prediction bound
- reportdict
Comprehensive diagnostics and method comparison
Usage Examples:
# Basic usage (auto-selects best method) L, U, report = compute_robust_prediction_bounds(loo_preds, n_test=50)
# Force conservative method L, U, report = compute_robust_prediction_bounds(
loo_preds, n_test=50, method=’exact’
)
# Compare all methods L, U, report = compute_robust_prediction_bounds(
loo_preds, n_test=50, method=’all’
) print(report[‘comparison_table’])
- ssbc.metrics.loo_uncertainty.format_prediction_bounds_report(rate_name, loo_predictions, n_test, alpha=0.05, include_all_methods=True)[source]
Generate a formatted text report of prediction bounds.
This produces human-readable output suitable for inclusion in rigorous analysis reports.
Parameters:
- rate_namestr
Name of the rate (e.g., ‘Singleton Rate’, ‘Doublet Rate’)
- loo_predictionsnp.ndarray
Binary LOO predictions
- n_testint
Test set size
- alphafloat
Significance level
- include_all_methodsbool
If True, compare all three methods in report
Returns:
- reportstr
Formatted text report