OCDocker.OCScore.Analysis.Metrics.Calibration module

Probability calibration metrics and post-hoc calibrators for OCScore classifiers.

Fits Platt scaling or isotonic regression on training/validation logits only, then reports Brier score, log loss, and expected calibration error (ECE) on evaluation splits.

OCDocker.OCScore.Analysis.Metrics.Calibration.LOGIT_CLIP = 50.0

Copyright (c) Federal University of Rio de Janeiro (UFRJ), Artur Duque Rossi, and Pedro Henrique Monteiro Torres.

SPDX-License-Identifier: BSD-3-Clause

See the LICENSE file for full terms.

OCDocker.OCScore.Analysis.Metrics.Calibration.logits_to_probabilities(logits)[source]

Map classifier logits to probabilities with a numerically stable sigmoid.

Parameters:

logits (np.ndarray) – One-dimensional classifier logits.

Returns:

Probabilities in (0, 1) with clipping applied for stability.

Return type:

np.ndarray

OCDocker.OCScore.Analysis.Metrics.Calibration.clip_probabilities(probabilities)[source]

Clip probabilities into the open unit interval for log loss.

Parameters:

probabilities (np.ndarray) – Raw predicted probabilities.

Returns:

Probabilities clipped to (eps, 1 - eps).

Return type:

np.ndarray

OCDocker.OCScore.Analysis.Metrics.Calibration.expected_calibration_error(y_true, y_prob, *, n_bins=10)[source]

Compute expected calibration error with uniform probability bins.

Parameters:

  • y_true (np.ndarray) – Binary ground-truth labels (0/1).

  • y_prob (np.ndarray) – Predicted probabilities for the positive class.

  • n_bins (int, optional) – Number of uniform bins on [0, 1], by default 10.

Returns:

Weighted mean absolute difference between bin accuracy and confidence, or nan when the input is empty or single-class.

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Calibration.evaluate_calibration_metrics(y_true, y_prob, *, n_bins=10)[source]

Return Brier score, log loss, and ECE for probabilistic predictions.

Parameters:

  • y_true (np.ndarray) – Binary ground-truth labels (0/1).

  • y_prob (np.ndarray) – Predicted probabilities for the positive class.

  • n_bins (int, optional) – Bin count for ECE, by default 10.

Returns:

Mapping with keys "Brier", "Log-loss", and "ECE". Values are nan when the input is empty or single-class.

Return type:

dict[str, float]

OCDocker.OCScore.Analysis.Metrics.Calibration.reliability_curve_points(y_true, y_prob, *, n_bins=10)[source]

Return mean predicted probability and fraction of positives per bin.

Parameters:

  • y_true (np.ndarray) – Binary ground-truth labels (0/1).

  • y_prob (np.ndarray) – Predicted probabilities for the positive class.

  • n_bins (int, optional) – Number of calibration bins, by default 10.

Returns:

(mean_predicted, fraction_positives) per bin; empty arrays when calibration cannot be computed.

Return type:

tuple[np.ndarray, np.ndarray]

class OCDocker.OCScore.Analysis.Metrics.Calibration.ProbabilityCalibrator(method, scores_are_logits=True, _platt=None, _isotonic=None)[source]

Bases: object

Post-hoc probability calibrator fit on one split and applied to others.

Parameters:

  • method (CalibrationMethod) – Calibration method: "platt" or "isotonic".

  • scores_are_logits (bool, optional) – If True, scores are raw logits; otherwise they are probabilities, by default True.

  • _platt (LogisticRegression | None)

  • _isotonic (IsotonicRegression | None)

method: Literal['platt', 'isotonic']
scores_are_logits: bool = True
classmethod fit(y_true, scores, *, method='platt', scores_are_logits=True)[source]

Fit Platt or isotonic calibration on reference labels and scores.

Parameters:

  • y_true (np.ndarray) – Binary ground-truth labels (0/1) from the fit split only.

  • scores (np.ndarray) – Model scores or logits aligned with y_true.

  • method (CalibrationMethod, optional) – Calibration method, by default "platt".

  • scores_are_logits (bool, optional) – Whether scores are logits, by default True.

Returns:

Fitted calibrator ready for predict().

Return type:

ProbabilityCalibrator

Raises:

ValueError – If y_true is single-class or method is unsupported.

predict(scores)[source]

Return calibrated probabilities for new scores or logits.

Parameters:

scores (np.ndarray) – Model scores or logits for the evaluation split.

Returns:

Calibrated probabilities in (0, 1).

Return type:

np.ndarray

Raises:

RuntimeError – If fit() has not been called.

to_dict()[source]

Serialize calibrator metadata for JSON export.

Returns:

JSON-serializable method name and scores_are_logits flag.

Return type:

dict[str, Any]

OCDocker.OCScore.Analysis.Metrics.Calibration.is_calibration_metric_key(key)[source]

Return True when key names a calibration export metric.

Parameters:

key (str)

Return type:

bool

OCDocker.OCScore.Analysis.Metrics.Calibration.apply_calibration_report_mode_to_metrics(metrics, mode='ranking_only')[source]

Rename calibration keys with diagnostic_ when reporting ranking-only claims.

Parameters:

  • metrics (dict[str, Any]) – Metrics dictionary updated in place.

  • mode (CalibrationReportMode, optional) – Report mode; ranking_only prefixes calibration keys.

Returns:

The same metrics dict.

Return type:

dict[str, Any]

OCDocker.OCScore.Analysis.Metrics.Calibration.collect_calibration_report_issues(metrics, mode='ranking_only')[source]

Return human-readable issues when calibration keys violate report mode.

Parameters:

  • metrics (Mapping[str, Any])

  • mode (Literal['ranking_only', 'calibration_validated'])

Return type:

list[str]

OCDocker.OCScore.Analysis.Metrics.Calibration.validate_calibration_report_mode(metrics, mode='ranking_only', *, strict=False)[source]

Validate calibration metric naming for the selected report mode.

Raises:

ValueError – When strict is True and ranking-only violations are found.

Parameters:

  • metrics (Mapping[str, Any])

  • mode (Literal['ranking_only', 'calibration_validated'])

  • strict (bool)

Return type:

list[str]

OCDocker.OCScore.Analysis.Metrics.Calibration.extract_calibration_metrics(metrics, mode='ranking_only')[source]

Return calibration-related entries from a metrics mapping.

Parameters:

  • metrics (Mapping[str, Any])

  • mode (Literal['ranking_only', 'calibration_validated'])

Return type:

dict[str, Any]

OCDocker.OCScore.Analysis.Metrics.Calibration.build_calibration_report_section(validation_metrics, test_metrics, *, mode='ranking_only', calibrator=None, val_true=None, val_scores=None)[source]

Build a JSON-friendly calibration subsection for production-grade reports.

Parameters:

  • validation_metrics (Mapping[str, Any])

  • test_metrics (Mapping[str, Any])

  • mode (Literal['ranking_only', 'calibration_validated'])

  • calibrator (ProbabilityCalibrator | None)

  • val_true (ndarray | None)

  • val_scores (ndarray | None)

Return type:

dict[str, Any]

OCDocker.OCScore.Analysis.Metrics.Calibration.merge_calibration_metrics(metrics, y_true, scores, *, calibrator=None, scores_are_logits=True, n_bins=10, include_uncalibrated=True, include_calibrated=True)[source]

Add calibration metrics to an existing metrics mapping (in place).

Parameters:

  • metrics (dict[str, Any]) – Metrics dictionary updated in place.

  • y_true (np.ndarray) – Binary ground-truth labels.

  • scores (np.ndarray) – Model scores or logits aligned with y_true.

  • calibrator (ProbabilityCalibrator, optional) – Fitted calibrator for calibrated metrics, by default None.

  • scores_are_logits (bool, optional) – Whether scores are logits when no calibrator is supplied, by default True.

  • n_bins (int, optional) – Bin count for ECE, by default 10.

  • include_uncalibrated (bool, optional) – Write uncalibrated Brier/log-loss/ECE keys, by default True.

  • include_calibrated (bool, optional) – Write *_calibrated keys when calibrator is set, by default True.

Returns:

The same metrics dict, updated in place.

Return type:

dict[str, Any]

OCDocker.OCScore.Analysis.Metrics.Calibration.enrich_dudez_export_metrics(validation_metrics, test_metrics, *, val_true, val_scores, test_true, test_scores, calibration_method='platt', report_mode='ranking_only')[source]

Fit calibration on validation logits and enrich val/test metric dicts.

Parameters:

  • validation_metrics (dict[str, Any]) – Validation metrics dict updated in place.

  • test_metrics (dict[str, Any]) – Test metrics dict updated in place.

  • val_true (np.ndarray) – Validation labels and logits used to fit the calibrator.

  • val_scores (np.ndarray) – Validation labels and logits used to fit the calibrator.

  • test_true (np.ndarray) – Test labels and logits used for calibrated test metrics.

  • test_scores (np.ndarray) – Test labels and logits used for calibrated test metrics.

  • calibration_method (CalibrationMethod, optional) – Calibration method, by default "platt".

  • report_mode (CalibrationReportMode, optional) – Controls whether calibration keys are prefixed as diagnostic-only.

Returns:

Fitted calibrator applied to both splits.

Return type:

ProbabilityCalibrator

OCDocker.OCScore.Analysis.Metrics.Calibration.calibration_metric_names(*, include_calibrated=True)[source]

Return calibration metric column names for CSV/JSON exports.

Parameters:

include_calibrated (bool, optional) – Include *_calibrated suffix variants, by default True.

Returns:

Ordered metric key names.

Return type:

tuple[str, …]