ssbc.reporting

Reporting and visualization APIs.

Re-export reporting and visualization helpers under a dedicated namespace.

ssbc.reporting.generate_rigorous_pac_report(labels, probs, alpha_target=0.1, delta=0.1, test_size=None, ci_level=0.95, use_union_bound=False, n_jobs=-1, verbose=True, prediction_method='exact', use_loo_correction=True, loo_inflation_factor=None)[source]

Generate complete rigorous PAC report with coverage volatility.

This is the UNIFIED function that gives you everything properly: - SSBC-corrected thresholds - Coverage guarantees - PAC-controlled operational bounds (marginal + per-class) - Singleton error rates with PAC guarantees - All bounds account for coverage volatility via BetaBinomial

Parameters:

  • labels (np.ndarray, shape (n,)) – True labels (0 or 1)

  • probs (np.ndarray, shape (n, 2)) – Predicted probabilities [P(class=0), P(class=1)]

  • alpha_target (float or dict[int, float], default=0.10) – Target miscoverage per class

  • delta (float or dict[int, float], default=0.10) – PAC risk tolerance. Used for both: - Coverage guarantee (via SSBC) - Operational bounds (pac_level = 1 - delta)

  • test_size (int, optional) – Expected test set size. If None, uses calibration size

  • ci_level (float, default=0.95) – Confidence level for prediction bounds

  • prediction_method (str, default="hoeffding") – Method for LOO uncertainty quantification (when use_loo_correction=True): - “auto”: Automatically select best method - “analytical”: Method 1 (recommended for n>=40) - “exact”: Method 2 (recommended for n=20-40) - “hoeffding”: Method 3 (ultra-conservative, default) - “all”: Compare all methods When use_loo_correction=False, this parameter is ignored.

  • use_loo_correction (bool, default=False) –

    If True, uses LOO-CV uncertainty correction for small samples (n=20-40). This accounts for all four sources of uncertainty: 1. LOO-CV correlation structure (variance inflation ≈2×) 2. Threshold calibration uncertainty 3. Parameter estimation uncertainty 4. Test sampling uncertainty Recommended for small calibration sets where standard bounds may be too narrow.

    LOO-CV Correlation Issue: The critical challenge with LOO-CV is that the N LOO predictions are not independent. The training sets for different folds overlap substantially—folds i and j using training sets D_{-i} and D_{-j} differ by only two examples out of N−1. Because each fold’s threshold is computed from nearly identical data, the resulting predictions exhibit strong positive correlation. This correlation structure is handled through specialized LOO-corrected methods that account for the dependency between folds when computing diagnostic bounds.

  • loo_inflation_factor (float, optional) –

    Manual override for LOO variance inflation factor. If None (default), automatically estimated from the data using empirical variance.

    Empirical Correction Factor Estimation: The inflation factor is estimated by comparing the empirical variance of LOO predictions to the theoretical IID variance. Specifically, inflation = (Var_empirical / Var_IID) × (n / (n-1)), where Var_empirical is the sample variance of the binary LOO predictions (with Bessel’s correction), Var_IID = p̂(1-p̂) is the expected variance under independence, and the n/(n-1) factor accounts for the finite-sample bias correction. For large n, this approaches the theoretical value of 2.0, but for small samples (n=20-40), the actual inflation can vary. The estimated factor is clipped to [1.0, 6.0] to prevent extreme values from outliers or numerical instability.

    Typical values: - 1.0: No inflation (assumes independent samples - usually wrong for LOO) - 2.0: Standard LOO inflation (theoretical value for n→∞) - 1.5-2.5: Empirical range for small samples - >2.5: High correlation scenarios - Up to 6.0: Extended range for very high correlation scenarios

    Note: This parameter can be used as a phenomenological control knob to correct for issues not modeled properly in the statistical framework. For example, if validation suggests the default estimation is too optimistic or too conservative, manually adjusting this factor can help achieve desired coverage behavior. Use with caution and validate empirically.

  • use_union_bound (bool, default=False) – Apply Bonferroni for simultaneous guarantees

  • n_jobs (int, default=-1) – Number of parallel jobs for LOO-CV computation. -1 = use all cores (default), 1 = single-threaded, N = use N cores.

  • verbose (bool, default=True) – Print comprehensive report

Returns:

Complete report with keys: - ‘ssbc_class_0’: SSBCResult for class 0 - ‘ssbc_class_1’: SSBCResult for class 1 - ‘pac_bounds_marginal’: PAC operational bounds (marginal) - ‘pac_bounds_class_0’: PAC operational bounds (class 0) - ‘pac_bounds_class_1’: PAC operational bounds (class 1) - ‘calibration_result’: From mondrian_conformal_calibrate - ‘prediction_stats’: From mondrian_conformal_calibrate

Return type:

dict

Examples

>>> from ssbc import BinaryClassifierSimulator
>>> from ssbc.rigorous_report import generate_rigorous_pac_report
>>>
>>> sim = BinaryClassifierSimulator(p_class1=0.5, seed=42)
>>> labels, probs = sim.generate(n_samples=1000)
>>>
>>> report = generate_rigorous_pac_report(
...     labels, probs,
...     alpha_target=0.10,
...     delta=0.10,
...     verbose=True
... )

Notes

This replaces the old workflow (removed in v1.1.0):

OLD (removed - these functions no longer exist): `python # These functions were removed in v1.1.0: # op_bounds = compute_mondrian_operational_bounds(...)  # Removed # marginal_bounds = compute_marginal_operational_bounds(...)  # Removed # report_prediction_stats(...)  # Removed `

NEW (rigorous): `python report = generate_rigorous_pac_report(labels, probs, alpha_target, delta) # Done! All bounds account for coverage volatility. `

ssbc.reporting.plot_parallel_coordinates_plotly(df, columns=None, color='err_all', color_continuous_scale=None, title='Mondrian sweep – interactive parallel coordinates', height=600, base_opacity=0.9, unselected_opacity=0.06)[source]

Create interactive parallel coordinates plot for hyperparameter sweep results.

Parameters:

  • df (pd.DataFrame) – DataFrame with hyperparameter sweep results

  • columns (list of str, optional) – Columns to display in parallel coordinates Default: [‘a0’,’d0’,’a1’,’d1’,’cov’,’sing_rate’,’err_all’,’err_pred0’,’err_pred1’,’err_y0’,’err_y1’,’esc_rate’]

  • color (str, default='err_all') – Column to use for coloring lines

  • color_continuous_scale (plotly colorscale, optional) – Color scale for the lines

  • title (str, default="Mondrian sweep – interactive parallel coordinates") – Plot title

  • height (int, default=600) – Plot height in pixels

  • base_opacity (float, default=0.9) – Opacity of selected lines

  • unselected_opacity (float, default=0.06) – Opacity of unselected lines (creates contrast)

Returns:

Interactive plotly figure

Return type:

plotly.graph_objects.Figure

Examples

>>> import pandas as pd
>>> df = sweep_hyperparams_and_collect(...)
>>> fig = plot_parallel_coordinates_plotly(df, color='err_all')
>>> fig.show()  # In notebook
>>> # Or save: fig.write_html("sweep_results.html")
ssbc.reporting.report_prediction_stats(prediction_stats, calibration_result, operational_bounds_per_class=None, marginal_operational_bounds=None, verbose=True)[source]

Report rigorous statistics for Mondrian conformal prediction with valid CIs.

Only displays statistics with valid confidence intervals: - Per-class statistics from calibration data (valid within class) - Per-class operational bounds from cross-validation (rigorous PAC bounds) - Marginal operational bounds from cross-validated Mondrian (rigorous PAC bounds)

Does NOT display marginal statistics from calibration data (invalid CIs for Mondrian).

Parameters:

  • prediction_stats (dict) – Output from mondrian_conformal_calibrate (second return value)

  • calibration_result (dict) – Output from mondrian_conformal_calibrate (first return value)

  • operational_bounds_per_class (dict[int, OperationalRateBoundsResult], optional) – Per-class operational bounds (from generate_rigorous_pac_report)

  • marginal_operational_bounds (OperationalRateBoundsResult, optional) – Marginal operational bounds (from generate_rigorous_pac_report)

  • verbose (bool, default=True) – If True, print detailed statistics to stdout

Returns:

Structured summary with valid CIs: - Keys 0, 1 for per-class statistics - Key ‘marginal_bounds’ if marginal_operational_bounds provided

Return type:

dict

Examples

>>> # Get operational bounds from rigorous PAC report
>>> from ssbc import generate_rigorous_pac_report
>>> report = generate_rigorous_pac_report(labels, probs, alpha_target=0.10, delta=0.10)
>>> cal_result = report['calibration_result']
>>> pred_stats = report['prediction_stats']
>>> op_bounds = report['pac_bounds_class_0']  # Per-class bounds
>>> marginal = report['pac_bounds_marginal']  # Marginal bounds
>>> summary = report_prediction_stats(pred_stats, cal_result, op_bounds, marginal)

Modules

rigorous_report

Unified rigorous reporting with full PAC guarantees.

visualization

Visualization and reporting utilities for conformal prediction results.