ssbc.calibration.cross_conformal
Cross-conformal validation for estimating rate variability.
This implements K-fold cross-validation specifically for conformal prediction: - Split calibration data into K folds - For each fold: train thresholds on K-1 folds, evaluate rates on held-out fold - Aggregate rates across folds to quantify finite-sample variability
Different from: - LOO-CV: Leave-one-out, aggregates counts (not rates per fold) - Bootstrap: Resamples with replacement, tests on fresh data - Cross-conformal: K-fold split, estimates rate distribution from finite calibration
Functions
|
K-fold cross-conformal validation for Mondrian conformal prediction. |
|
Pretty print cross-conformal validation results. |
- ssbc.calibration.cross_conformal.cross_conformal_validation(labels, probs, alpha_target=0.1, delta=0.1, n_folds=5, stratify=True, seed=None)[source]
K-fold cross-conformal validation for Mondrian conformal prediction.
Estimates the variability of operational rates (abstentions, singletons, doublets) due to finite calibration sample effects by splitting data into K folds.
For each fold: 1. Train: Compute SSBC-corrected thresholds on K-1 folds 2. Test: Evaluate operational rates on held-out fold 3. Record: Store rates for this fold
Aggregate rates across folds to quantify finite-sample variability.
- Parameters:
labels (np.ndarray, shape (n,)) – Calibration labels (0 or 1)
probs (np.ndarray, shape (n, 2)) – Calibration probabilities [P(class=0), P(class=1)]
alpha_target (float, default=0.10) – Target miscoverage rate
delta (float, default=0.10) – PAC risk for SSBC correction
n_folds (int, default=5) – Number of folds (K)
stratify (bool, default=True) – Stratify folds by class labels
seed (int, optional) – Random seed for reproducibility
- Returns:
Cross-conformal results with keys: - ‘fold_rates’: List of rate dicts for each fold - ‘marginal’: Statistics for marginal rates - ‘class_0’: Statistics for class 0 rates - ‘class_1’: Statistics for class 1 rates Each statistics dict contains: - ‘samples’: Array of rates across folds - ‘mean’: Mean rate - ‘std’: Standard deviation - ‘quantiles’: Dict with q05, q25, q50, q75, q95 - ‘ci_95’: 95% Clopper-Pearson CI (if applicable)
- Return type:
Examples
>>> from ssbc import cross_conformal_validation >>> results = cross_conformal_validation(labels, probs, n_folds=10) >>> m = results['marginal']['singleton'] >>> print(f"Singleton rate: {m['mean']:.3f} ± {m['std']:.3f}") >>> print(f"95% range: [{m['quantiles']['q05']:.3f}, {m['quantiles']['q95']:.3f}]")
Notes
Different from other methods: - LOO-CV: Leave-one-out, aggregates counts (not fold-level rates) - Bootstrap: Resamples with replacement, tests on fresh data - Cross-conformal: K-fold split, estimates rate distribution from calibration
This method directly estimates the variability of rates due to finite calibration samples, without requiring a data simulator.