OCDocker.OCScore.Analysis.Metrics.Ranking module

Core ranking metrics used across Analysis (ROC AUC, PR AUC, EF, BEDROC, etc.).

Usage:

from OCDocker.OCScore.Analysis.Metrics import Ranking as Rank

OCDocker.OCScore.Analysis.Metrics.Ranking.DEFAULT_SCREENING_COMPARISON_METRICS = ('BEDROC', 'ROC-AUC', 'PR-AUC', 'EF1%', 'EF5%', 'NDCG@1%', 'NDCG@5%', 'Precision', 'Recall', 'F1', 'MCC', 'TP', 'FP', 'TN', 'FN')

Copyright (c) Federal University of Rio de Janeiro (UFRJ), Artur Duque Rossi, and Pedro Henrique Monteiro Torres.

SPDX-License-Identifier: BSD-3-Clause

See the LICENSE file for full terms.

OCDocker.OCScore.Analysis.Metrics.Ranking.orient_scores(y_score, higher_is_better=True)[source]

Orient scores so that larger values indicate a better active candidate.

Ranking helpers in this module assume higher_is_better=True. Classifier logits and probabilities should be passed unchanged. Lower-is-better docking scores must be negated before ranking.

Parameters:

  • y_score (np.ndarray) – Raw target scores.

  • higher_is_better (bool, optional) – Whether larger raw scores indicate better actives, by default True.

Returns:

Scores oriented for descending ranking.

Return type:

np.ndarray

OCDocker.OCScore.Analysis.Metrics.Ranking.is_valid_ranking_scores(y_score, epsilon=1e-08)[source]

Return whether scores provide a meaningful ranking for early-enrichment metrics.

Constant or near-constant scores are invalid because tie-breaking would depend on row order rather than model quality.

Parameters:

  • y_score (np.ndarray) – Oriented screening scores where larger is better.

  • epsilon (float, optional) – Minimum standard deviation required for a valid ranking.

Returns:

True when at least two unique finite scores exist and std >= epsilon.

Return type:

bool

OCDocker.OCScore.Analysis.Metrics.Ranking.score_ranking_diagnostics(y_score, epsilon=1e-08)[source]

Summarize score distribution for ranking-metric validity checks.

Parameters:

  • y_score (np.ndarray) – Oriented screening scores.

  • epsilon (float, optional) – Minimum standard deviation required for a valid ranking.

Returns:

Diagnostics including score_std, n_unique_scores, min_score, max_score, and ranking_valid.

Return type:

dict[str, float]

OCDocker.OCScore.Analysis.Metrics.Ranking.bedroc(y_true, y_score, alpha=20.0)[source]

BEDROC per Truchon & Bayly (2007), ranking by descending score.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores where larger values indicate better actives.

  • alpha (float, optional) – Exponential weighting factor; higher = more early recognition. Default is 20.0.

Returns:

BEDROC score (0.0 ~ 1.0, or NaN if no positives).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.rie(y_true, y_score, alpha=20.0)[source]

Robust Initial Enhancement (RIE) for early recognition.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores where larger values indicate better actives.

  • alpha (float, optional) – Exponential weighting factor, by default 20.0.

Returns:

RIE score, or NaN when there are no positives.

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.aggregate_group_metric(y_true, y_score, groups, metric_fn, epsilon=1e-08)[source]

Aggregate one ranking metric as the mean over valid groups.

Groups with fewer than two classes, zero actives, zero decoys, or invalid constant/tied scores are skipped instead of contributing invalid values.

Parameters:

  • y_true (np.ndarray) – Binary labels.

  • y_score (np.ndarray) – Oriented scores where larger is better.

  • groups (Iterable) – Group label per sample.

  • metric_fn (Callable[[np.ndarray, np.ndarray], float]) – Metric function accepting (y_true, y_score).

  • epsilon (float, optional) – Minimum within-group score standard deviation for ranking validity.

Returns:

Mean metric across valid groups, number of valid groups, total unique groups observed, and per-reason skip counts.

Return type:

tuple[float, int, int, dict[str, int]]

OCDocker.OCScore.Analysis.Metrics.Ranking.evaluate_screening_metrics_by_group(y_true, y_score, groups, *, higher_is_better=True, metric_names=None, bedroc_alpha=20.0)[source]

Evaluate screening metrics separately for each receptor/group.

Parameters:

  • y_true (np.ndarray) – Binary labels.

  • y_score (np.ndarray) – Raw scores (oriented via higher_is_better).

  • groups (Iterable) – Group label per sample.

  • higher_is_better (bool, optional) – Whether larger raw scores favor actives, by default True.

  • metric_names (Sequence[str] | None, optional) – Metrics to compute per group. Defaults to BEDROC, ROC-AUC, PR-AUC, EF, and NDCG variants.

  • bedroc_alpha (float, optional) – Exponential BEDROC weighting factor for per-group BEDROC, by default 20.0.

Returns:

One row per valid group with a group column and metric columns.

Return type:

pandas.DataFrame

OCDocker.OCScore.Analysis.Metrics.Ranking.evaluate_scoring_functions_by_group(dataframe, validation_indices, labels, groups, columns, *, metric_names, column_higher_is_better=None, bedroc_alpha=20.0)[source]

Evaluate scoring-function columns with per-group screening metrics.

Parameters:

  • dataframe (pandas.DataFrame) – Full reduced dataframe containing scoring-function columns.

  • validation_indices (np.ndarray) – Row indices for the validation fold.

  • labels (np.ndarray) – Binary labels aligned with dataframe rows.

  • groups (np.ndarray) – Receptor/group labels for validation rows.

  • columns (Sequence[str]) – Scoring-function column names.

  • metric_names (Sequence[str]) – Metrics to retain in the output.

  • column_higher_is_better (Mapping[str, bool] | None, optional) – Per-column score orientation. Missing columns default to True.

  • bedroc_alpha (float, optional) – Exponential BEDROC weighting factor for per-group BEDROC, by default 20.0.

Returns:

Long-format rows with group, scorer, scorer_type, and metrics.

Return type:

pandas.DataFrame

OCDocker.OCScore.Analysis.Metrics.Ranking.enrichment_factor(y_true, y_score, fraction)[source]

EF@fraction (e.g., 0.01 for 1%). EF = hits_in_top_fraction / expected_hits_random.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores where larger values indicate better actives.

  • fraction (float) – Fraction of top-scoring samples to consider (0.0 ~ 1.0).

Returns:

Enrichment factor (>= 0.0, or NaN if no positives).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.ndcg_at_fraction(y_true, y_score, fraction)[source]

Compute NDCG at the top fraction of a ranked list.

Parameters:

  • y_true (np.ndarray) – True binary labels.

  • y_score (np.ndarray) – Oriented screening scores where larger is better.

  • fraction (float) – Top-ranked fraction to evaluate.

Returns:

NDCG score at the requested fraction, or NaN when ranking is invalid.

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.groupwise(y_true, y_score, groups)[source]

Compute macro/micro ROC/PR AUC across discrete groups.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

  • groups (Iterable) – Group labels for each sample (same length as y_true/y_score).

Returns:

Dictionary with keys “roc_auc_macro”, “pr_auc_macro”, “roc_auc_micro”, “pr_auc_micro” and corresponding float values (or NaN if undefined).

Return type:

dict[str, float]

OCDocker.OCScore.Analysis.Metrics.Ranking.pr_auc(y_true, y_score)[source]

Compute average precision (area under PR curve).

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

Returns:

Average precision score (0.0 ~ 1.0).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.riep(y_true, y_score, k)[source]

Relative enrichment among the top-k versus total positives.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

  • k (int) – Number of top-scoring samples to consider.

Returns:

RIEP score (0.0 ~ 1.0, or NaN if no positives).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.roc_auc(y_true, y_score)[source]

Compute ROC AUC with defensive validation.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

Returns:

ROC AUC score (0.0 ~ 1.0).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.threshold_at_precision(y_true, y_score, target_precision)[source]

Find first threshold achieving at least the given precision.

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

  • target_precision (float) – Desired precision level (0.0 ~ 1.0).

Returns:

(threshold, precision, recall) at first point where precision >= target_precision, or (NaN, NaN, NaN) if target_precision not achievable.

Return type:

tuple(float, float, float)

OCDocker.OCScore.Analysis.Metrics.Ranking.top_fraction_precision(y_true, y_score, frac)[source]

Precision among the top fraction (e.g., 0.01 for top-1%).

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

  • frac (float) – Fraction of top-scoring samples to consider (0.0 ~ 1.0).

Returns:

Precision among top fraction (0.0 ~ 1.0).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.top_k_precision(y_true, y_score, k)[source]

Precision among the top-k scored samples (descending by score).

Parameters:

  • y_true (np.ndarray) – True binary labels (0/1 or boolean).

  • y_score (np.ndarray) – Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by a classifier).

  • k (int) – Number of top-scoring samples to consider.

Returns:

Precision among top-k (0.0 ~ 1.0).

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.binary_threshold_youden(y_true, y_score)[source]

Pick the score threshold that maximizes Youden’s J (TPR - FPR).

Parameters:

  • y_true (ndarray)

  • y_score (ndarray)

Return type:

float

OCDocker.OCScore.Analysis.Metrics.Ranking.confusion_counts(y_true, y_pred)[source]

Return TP/FP/TN/FN counts for binary labels and predictions.

Parameters:

  • y_true (ndarray)

  • y_pred (ndarray)

Return type:

dict[str, float]

OCDocker.OCScore.Analysis.Metrics.Ranking.classification_metrics_at_threshold(y_true, y_score, threshold=None)[source]

Compute threshold-based classification metrics and confusion counts.

When threshold is omitted, Youden’s J on the ROC curve selects the cutoff. y_score must already be oriented so larger values favor actives.

Parameters:

  • y_true (ndarray)

  • y_score (ndarray)

  • threshold (float | None)

Return type:

dict[str, float]

OCDocker.OCScore.Analysis.Metrics.Ranking.aggregate_group_classification_metrics(y_true, y_score, groups, *, metric_keys=('Precision', 'Recall', 'F1', 'MCC'))[source]

Macro-average classification metrics over groups with both classes.

Parameters:

  • y_true (ndarray)

  • y_score (ndarray)

  • groups (Iterable)

  • metric_keys (Sequence[str])

Return type:

Tuple[dict[str, float], int, int]

OCDocker.OCScore.Analysis.Metrics.Ranking.evaluate_screening_metrics(y_true, y_score, groups=None, higher_is_better=True, bedroc_alpha=20.0)[source]

Evaluate DUDEz classification and early-recognition metrics.

Classifier logits and probabilities should use higher_is_better=True. Lower-is-better docking scores must set higher_is_better=False.

When groups is provided, BEDROC, EF, and NDCG are averaged across targets/receptors with both actives and decoys present.

Parameters:

  • bedroc_alpha (float, optional) – Exponential BEDROC weighting factor, by default 20.0.

  • y_true (ndarray)

  • y_score (ndarray)

  • groups (ndarray | None)

  • higher_is_better (bool)

Return type:

dict[str, float]