ssbc.metrics
Operational metrics and uncertainty APIs.
This package provides functions for computing operational bounds, LOO-CV uncertainty quantification, and prediction bounds.
- ssbc.metrics.compute_pac_operational_bounds_marginal(ssbc_result_0, ssbc_result_1, labels, probs, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='simple')[source]
Compute marginal operational bounds for FIXED calibration via LOO-CV.
Enhanced approach: 1. Use FIXED u_star positions from SSBC calibration 2. Run LOO-CV to get unbiased rate estimates 3. Apply prediction bounds accounting for both calibration and test set sampling uncertainty 4. Optional union bound for simultaneous guarantees
This models: “Given fixed calibration, what are rate distributions on future test sets?” The prediction bounds account for both calibration uncertainty and test set sampling variability.
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
test_size (int) – Expected test set size for prediction bounds. Used to account for test set sampling uncertainty.
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
pac_level (float)
prediction_method (str)
- Returns:
Operational bounds with keys: - ‘singleton_rate_bounds’: [L, U] - ‘doublet_rate_bounds’: [L, U] - ‘abstention_rate_bounds’: [L, U] - ‘singleton_error_rate_class0_bounds’: [L, U] - ‘singleton_error_rate_class1_bounds’: [L, U] - ‘singleton_error_rate_cond_class0_bounds’: [L, U] - ‘singleton_error_rate_cond_class1_bounds’: [L, U] - ‘expected_*_rate’: point estimates
Note: Marginal singleton_error_rate_bounds is NOT computed because it mixes two different distributions (class 0 and class 1) which cannot be justified statistically.
- Return type:
- ssbc.metrics.compute_pac_operational_bounds_marginal_loo_corrected(ssbc_result_0, ssbc_result_1, labels, probs, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='auto', loo_inflation_factor=None, verbose=True)[source]
Compute marginal operational bounds with LOO-CV uncertainty correction.
This function uses the new LOO uncertainty quantification that properly accounts for all four sources of uncertainty: 1. LOO-CV correlation structure 2. Threshold calibration uncertainty 3. Parameter estimation uncertainty 4. Test sampling uncertainty
This is the RECOMMENDED function for small calibration sets (n=20-40).
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
test_size (int) – Expected test set size for prediction bounds
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
prediction_method (str, default="auto") – Method for LOO uncertainty quantification: - “auto”: Automatically select best method - “analytical”: Method 1 (recommended for n>=40) - “exact”: Method 2 (recommended for n=20-40) - “hoeffding”: Method 3 (ultra-conservative) - “all”: Compare all methods
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, automatically estimated. Typical values: 1.0 (no inflation), 2.0 (standard LOO), 1.5-2.5 (empirical range)
pac_level (float)
verbose (bool)
- Returns:
Operational bounds with keys: - ‘singleton_rate_bounds’: [L, U] - ‘doublet_rate_bounds’: [L, U] - ‘abstention_rate_bounds’: [L, U] - ‘singleton_error_rate_class0_bounds’: [L, U] (error when true_label=0) - ‘singleton_error_rate_class1_bounds’: [L, U] (error when true_label=1) - ‘singleton_correct_rate_class0_bounds’: [L, U] (correct when true_label=0) - ‘singleton_correct_rate_class1_bounds’: [L, U] (correct when true_label=1) - ‘singleton_error_rate_pred_class0_bounds’: [L, U] (error when predicted_class=0) - ‘singleton_error_rate_pred_class1_bounds’: [L, U] (error when predicted_class=1) - ‘singleton_correct_rate_pred_class0_bounds’: [L, U] (correct when predicted_class=0) - ‘singleton_correct_rate_pred_class1_bounds’: [L, U] (correct when predicted_class=1) - ‘expected_*_rate’: point estimates - ‘loo_diagnostics’: Detailed LOO uncertainty analysis
Note: Marginal singleton_error_rate_bounds is NOT computed because it mixes two different distributions (class 0 and class 1) which cannot be justified statistically. Note: Conditional rates are NOT computed in the marginal section.
- Return type:
- ssbc.metrics.compute_pac_operational_bounds_perclass(ssbc_result_0, ssbc_result_1, labels, probs, class_label, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='simple', loo_inflation_factor=None)[source]
Compute per-class operational bounds for FIXED calibration via LOO-CV.
- Parameters:
class_label (int) – Which class to analyze (0 or 1)
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, not used. Note: Per-class bounds currently use standard prediction bounds, not LOO-corrected bounds. This parameter is included for API compatibility and future use.
ssbc_result_0 (SSBCResult)
ssbc_result_1 (SSBCResult)
labels (ndarray)
probs (ndarray)
test_size (int)
ci_level (float)
pac_level (float)
use_union_bound (bool)
n_jobs (int)
prediction_method (str)
- Return type:
Notes
The test_size is automatically adjusted based on the expected class distribution: expected_n_class_test = test_size * (n_class_cal / n_total)
This ensures proper uncertainty quantification for class-specific rates.
Other parameters same as marginal version.
- Returns:
Per-class operational bounds
- Return type:
- Parameters:
- ssbc.metrics.compute_pac_operational_bounds_perclass_loo_corrected(ssbc_result_0, ssbc_result_1, labels, probs, class_label, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='auto', loo_inflation_factor=None, verbose=True)[source]
Compute per-class operational bounds with LOO-CV uncertainty correction.
This function uses LOO uncertainty quantification for per-class bounds, enabling method comparison (“all”) for individual classes.
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
class_label (int) – Which class to analyze (0 or 1)
test_size (int) – Expected test set size for prediction bounds
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
prediction_method (str, default="auto") – Method for LOO uncertainty quantification: - “auto”: Automatically select best method - “analytical”: Method 1 (recommended for n>=40) - “exact”: Method 2 (recommended for n=20-40) - “hoeffding”: Method 3 (ultra-conservative) - “all”: Compare all methods
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, automatically estimated.
verbose (bool, default=True) – Print diagnostic information
pac_level (float)
- Returns:
Per-class operational bounds with LOO diagnostics (when method=”all”)
- Return type:
- ssbc.metrics.estimate_loo_inflation_factor(loo_predictions, verbose=True)[source]
Estimate the actual variance inflation from LOO-CV for this specific problem.
Theory predicts inflation ≈ 2×, but the actual value depends on the model and data structure. This computes an empirical estimate.
Parameters:
- loo_predictionsnp.ndarray, shape (n_cal,)
Binary LOO predictions (1=success, 0=failure)
Returns:
- inflation_factorfloat
Estimated variance inflation, clipped to [1.0, 6.0] Typically ≈ 2.0 for LOO-CV
Notes:
Inflation = Var_empirical / Var_IID For n → ∞: inflation → 2.0 For small n: inflation can vary, so we clip to reasonable range
- ssbc.metrics.compute_loo_corrected_prediction_bounds(loo_predictions, n_test, alpha=0.05, method='simple', inflation_factor=None, verbose=True)[source]
Compute prediction bounds using Clopper-Pearson + sampling uncertainty with LOO correction.
This function provides the correct statistical approach: 1. Use Clopper-Pearson to bound the true rate p from calibration data 2. Add sampling uncertainty for future test sets of size n_test 3. Account for LOO-CV correlation structure
- Parameters:
loo_predictions (np.ndarray, shape (n_cal,)) – Binary LOO predictions (1=success, 0=failure)
n_test (int) – Expected size of future test sets
alpha (float) – Significance level (e.g., 0.05 for 95% confidence)
method (str) – ‘simple’ - Clopper-Pearson + normal approximation for sampling uncertainty ‘beta_binomial’ - Exact Beta-Binomial approach ‘hoeffding’ - Distribution-free Hoeffding bounds
inflation_factor (float, optional) – Manual override for LOO variance inflation factor
verbose (bool, default=True) – Print diagnostic information
- Returns:
(lower_bound, upper_bound, diagnostics)
- Return type:
- ssbc.metrics.compute_robust_prediction_bounds(loo_predictions, n_test, alpha=0.05, method='auto', inflation_factor=None, verbose=True)[source]
Main function: Compute robust prediction bounds for small-sample LOO-CV.
This is the primary entry point. It intelligently selects methods based on sample size and provides comprehensive diagnostics.
Parameters:
- loo_predictionsnp.ndarray, shape (n_cal,)
Binary LOO predictions (1=singleton/success, 0=not/failure)
- n_testint
Expected size of future test sets
- alphafloat
Significance level (e.g., 0.05 for 95% confidence)
- methodstr
‘auto’ - Automatically select best method (recommended) ‘analytical’ - Method 1: Analytical with LOO correction ‘exact’ - Method 2: Exact binomial with effective n ‘hoeffding’ - Method 3: Distribution-free bound ‘all’ - Compute all three and report
- inflation_factorfloat, optional
Manual override for LOO variance inflation factor. If None, automatically estimated. Typical values: 1.0 (no inflation), 2.0 (standard LOO), 1.5-2.5 (empirical range)
- verbosebool, default=True
If True, print diagnostic information about method selection and inflation factors.
Returns:
- L_primefloat
Lower prediction bound
- U_primefloat
Upper prediction bound
- reportdict
Comprehensive diagnostics and method comparison
Usage Examples:
# Basic usage (auto-selects best method) L, U, report = compute_robust_prediction_bounds(loo_preds, n_test=50)
# Force conservative method L, U, report = compute_robust_prediction_bounds(
loo_preds, n_test=50, method=’exact’
)
# Compare all methods L, U, report = compute_robust_prediction_bounds(
loo_preds, n_test=50, method=’all’
) print(report[‘comparison_table’])
- ssbc.metrics.format_prediction_bounds_report(rate_name, loo_predictions, n_test, alpha=0.05, include_all_methods=True)[source]
Generate a formatted text report of prediction bounds.
This produces human-readable output suitable for inclusion in rigorous analysis reports.
Parameters:
- rate_namestr
Name of the rate (e.g., ‘Singleton Rate’, ‘Doublet Rate’)
- loo_predictionsnp.ndarray
Binary LOO predictions
- n_testint
Test set size
- alphafloat
Significance level
- include_all_methodsbool
If True, compare all three methods in report
Returns:
- reportstr
Formatted text report
Modules
Small-sample uncertainty quantification for LOO-CV conformal prediction. |
|
Simplified operational bounds for fixed calibration (LOO-CV + CP). |