ssbc.metrics.operational_bounds_simple
Simplified operational bounds for fixed calibration (LOO-CV + CP).
Functions
Compute marginal operational bounds for FIXED calibration via LOO-CV. |
|
Compute marginal operational bounds with LOO-CV uncertainty correction. |
|
Compute per-class operational bounds for FIXED calibration via LOO-CV. |
|
Compute per-class operational bounds with LOO-CV uncertainty correction. |
- ssbc.metrics.operational_bounds_simple.compute_pac_operational_bounds_marginal(ssbc_result_0, ssbc_result_1, labels, probs, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='simple')[source]
Compute marginal operational bounds for FIXED calibration via LOO-CV.
Enhanced approach: 1. Use FIXED u_star positions from SSBC calibration 2. Run LOO-CV to get unbiased rate estimates 3. Apply prediction bounds accounting for both calibration and test set sampling uncertainty 4. Optional union bound for simultaneous guarantees
This models: “Given fixed calibration, what are rate distributions on future test sets?” The prediction bounds account for both calibration uncertainty and test set sampling variability.
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
test_size (int) – Expected test set size for prediction bounds. Used to account for test set sampling uncertainty.
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
pac_level (float)
prediction_method (str)
- Returns:
Operational bounds with keys: - ‘singleton_rate_bounds’: [L, U] - ‘doublet_rate_bounds’: [L, U] - ‘abstention_rate_bounds’: [L, U] - ‘singleton_error_rate_class0_bounds’: [L, U] - ‘singleton_error_rate_class1_bounds’: [L, U] - ‘singleton_error_rate_cond_class0_bounds’: [L, U] - ‘singleton_error_rate_cond_class1_bounds’: [L, U] - ‘expected_*_rate’: point estimates
Note: Marginal singleton_error_rate_bounds is NOT computed because it mixes two different distributions (class 0 and class 1) which cannot be justified statistically.
- Return type:
- ssbc.metrics.operational_bounds_simple.compute_pac_operational_bounds_marginal_loo_corrected(ssbc_result_0, ssbc_result_1, labels, probs, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='auto', loo_inflation_factor=None, verbose=True)[source]
Compute marginal operational bounds with LOO-CV uncertainty correction.
This function uses the new LOO uncertainty quantification that properly accounts for all four sources of uncertainty: 1. LOO-CV correlation structure 2. Threshold calibration uncertainty 3. Parameter estimation uncertainty 4. Test sampling uncertainty
This is the RECOMMENDED function for small calibration sets (n=20-40).
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
test_size (int) – Expected test set size for prediction bounds
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
prediction_method (str, default="auto") – Method for LOO uncertainty quantification: - “auto”: Automatically select best method - “analytical”: Method 1 (recommended for n>=40) - “exact”: Method 2 (recommended for n=20-40) - “hoeffding”: Method 3 (ultra-conservative) - “all”: Compare all methods
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, automatically estimated. Typical values: 1.0 (no inflation), 2.0 (standard LOO), 1.5-2.5 (empirical range)
pac_level (float)
verbose (bool)
- Returns:
Operational bounds with keys: - ‘singleton_rate_bounds’: [L, U] - ‘doublet_rate_bounds’: [L, U] - ‘abstention_rate_bounds’: [L, U] - ‘singleton_error_rate_class0_bounds’: [L, U] (error when true_label=0) - ‘singleton_error_rate_class1_bounds’: [L, U] (error when true_label=1) - ‘singleton_correct_rate_class0_bounds’: [L, U] (correct when true_label=0) - ‘singleton_correct_rate_class1_bounds’: [L, U] (correct when true_label=1) - ‘singleton_error_rate_pred_class0_bounds’: [L, U] (error when predicted_class=0) - ‘singleton_error_rate_pred_class1_bounds’: [L, U] (error when predicted_class=1) - ‘singleton_correct_rate_pred_class0_bounds’: [L, U] (correct when predicted_class=0) - ‘singleton_correct_rate_pred_class1_bounds’: [L, U] (correct when predicted_class=1) - ‘expected_*_rate’: point estimates - ‘loo_diagnostics’: Detailed LOO uncertainty analysis
Note: Marginal singleton_error_rate_bounds is NOT computed because it mixes two different distributions (class 0 and class 1) which cannot be justified statistically. Note: Conditional rates are NOT computed in the marginal section.
- Return type:
- ssbc.metrics.operational_bounds_simple.compute_pac_operational_bounds_perclass(ssbc_result_0, ssbc_result_1, labels, probs, class_label, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='simple', loo_inflation_factor=None)[source]
Compute per-class operational bounds for FIXED calibration via LOO-CV.
- Parameters:
class_label (int) – Which class to analyze (0 or 1)
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, not used. Note: Per-class bounds currently use standard prediction bounds, not LOO-corrected bounds. This parameter is included for API compatibility and future use.
ssbc_result_0 (SSBCResult)
ssbc_result_1 (SSBCResult)
labels (ndarray)
probs (ndarray)
test_size (int)
ci_level (float)
pac_level (float)
use_union_bound (bool)
n_jobs (int)
prediction_method (str)
- Return type:
Notes
The test_size is automatically adjusted based on the expected class distribution: expected_n_class_test = test_size * (n_class_cal / n_total)
This ensures proper uncertainty quantification for class-specific rates.
Other parameters same as marginal version.
- Returns:
Per-class operational bounds
- Return type:
- Parameters:
- ssbc.metrics.operational_bounds_simple.compute_pac_operational_bounds_perclass_loo_corrected(ssbc_result_0, ssbc_result_1, labels, probs, class_label, test_size, ci_level=0.95, pac_level=0.95, use_union_bound=True, n_jobs=-1, prediction_method='auto', loo_inflation_factor=None, verbose=True)[source]
Compute per-class operational bounds with LOO-CV uncertainty correction.
This function uses LOO uncertainty quantification for per-class bounds, enabling method comparison (“all”) for individual classes.
- Parameters:
ssbc_result_0 (SSBCResult) – SSBC result for class 0
ssbc_result_1 (SSBCResult) – SSBC result for class 1
labels (np.ndarray) – True labels
probs (np.ndarray) – Predicted probabilities
class_label (int) – Which class to analyze (0 or 1)
test_size (int) – Expected test set size for prediction bounds
ci_level (float, default=0.95) – Confidence level for prediction bounds
use_union_bound (bool, default=True) – Apply Bonferroni for simultaneous guarantees
n_jobs (int, default=-1) – Number of parallel jobs (-1 = all cores)
prediction_method (str, default="auto") – Method for LOO uncertainty quantification: - “auto”: Automatically select best method - “analytical”: Method 1 (recommended for n>=40) - “exact”: Method 2 (recommended for n=20-40) - “hoeffding”: Method 3 (ultra-conservative) - “all”: Compare all methods
loo_inflation_factor (float, optional) – Manual override for LOO variance inflation factor. If None, automatically estimated.
verbose (bool, default=True) – Print diagnostic information
pac_level (float)
- Returns:
Per-class operational bounds with LOO diagnostics (when method=”all”)
- Return type: