OCDocker.OCScore.Optimization.ModelCrossValidation module¶

K-fold cross-validation for exported OCScore PDBbind and DUDEz models.

Uses fixed hyperparameters from an exported best_model/ bundle and retrains a fresh model on each fold. DUDEz defaults to receptor-grouped folds so validation receptors are never seen during training, matching staged screening evaluation.

class OCDocker.OCScore.Optimization.ModelCrossValidation.CrossValidationConfig(n_folds=5, epochs=100, random_seed=42, shuffle=True, strategy='auto', group_column='receptor', kind_column='kind', include_scoring_function_baselines=True, include_descriptor_aggregate_baselines=True, include_sf_consensus_baselines=True, report_entity_overlap=True, entity_columns=('name', 'ligand_name', 'smiles'), include_calibration_metrics=True, calibration_method='platt')[source]¶

Bases: object

Configuration for exported-model cross-validation.

Parameters:

n_folds (int, optional) – Number of cross-validation folds, by default 5.
epochs (int, optional) – Training epochs per fold, by default 100.
random_seed (int, optional) – Seed for fold shuffling and weight initialization, by default 42.
shuffle (bool, optional) – Shuffle fold assignments, by default True.
strategy (str, optional) – auto, receptor_grouped, or row_kfold. auto uses receptor-grouped folds for DUDEz when group_column is present.
group_column (str, optional) – Receptor/target column for grouped DUDEz CV, by default "receptor".
kind_column (str, optional) – DUDEz kind column for label derivation, by default "kind".
include_scoring_function_baselines (bool, optional) – For DUDEz screening exports, also evaluate every post-filter scoring-function column on each fold’s validation split, by default True.
include_descriptor_aggregate_baselines (bool, optional) – For DUDEz screening exports, also evaluate row-wise mean/median/max/min over model input features on each validation fold (desc_* scorers), by default True.
include_sf_consensus_baselines (bool, optional) – For DUDEz screening exports, also evaluate row-wise mean/median/max/min across scoring-function columns only (sf_* scorers), by default True.
report_entity_overlap (bool, optional) – When True, record train/validation duplicate entity keys in fold diagnostics and log a warning, by default True.
entity_columns (tuple of str, optional) – Columns checked for duplicate entities across train and validation splits.
include_calibration_metrics (bool, optional) – Compute calibration metrics on validation folds, by default True.
calibration_method (str, optional) – Calibration method name (e.g. "platt"), by default "platt".

n_folds: int = 5¶

epochs: int = 100¶

random_seed: int = 42¶

shuffle: bool = True¶

strategy: str = 'auto'¶

group_column: str = 'receptor'¶

kind_column: str = 'kind'¶

include_scoring_function_baselines: bool = True¶

include_descriptor_aggregate_baselines: bool = True¶

include_sf_consensus_baselines: bool = True¶

report_entity_overlap: bool = True¶

entity_columns: tuple[str, ...] = ('name', 'ligand_name', 'smiles')¶

include_calibration_metrics: bool = True¶

calibration_method: str = 'platt'¶

class OCDocker.OCScore.Optimization.ModelCrossValidation.CrossValidationFoldResult(fold_index, n_train, n_validation, train_indices, validation_indices, validation_metrics, scoring_function_metrics=<factory>, per_target_metrics=<factory>, diagnostics=<factory>)[source]¶

Bases: object

Metrics and metadata for one cross-validation fold.

Parameters:

fold_index (int) – Zero-based fold index.
n_train (int) – Number of training rows in this fold.
n_validation (int) – Number of validation rows in this fold.
train_indices (list of int) – Row indices used for training.
validation_indices (list of int) – Row indices used for validation.
validation_metrics (dict) – Primary model metrics on the validation split.
scoring_function_metrics (dict, optional) – Per-scoring-function metrics on validation, by default empty dict.
per_target_metrics (list of dict, optional) – Per-target breakdown when applicable, by default empty list.
diagnostics (dict, optional) – Fold-level integrity and overlap diagnostics, by default empty dict.

fold_index: int¶

n_train: int¶

n_validation: int¶

train_indices: list[int]¶

validation_indices: list[int]¶

validation_metrics: dict[str, float]¶

scoring_function_metrics: dict[str, dict[str, float]]¶

per_target_metrics: list[dict[str, Any]]¶

diagnostics: dict[str, Any]¶

class OCDocker.OCScore.Optimization.ModelCrossValidation.CrossValidationResult(export_dir, task, n_folds, effective_folds, strategy, epochs, random_seed, objective_metric, fold_results, aggregate_validation_metrics, model_config, scoring_function_columns=<factory>, aggregate_scoring_function_metrics=<factory>, scorer_comparison_summary=<factory>, diagnostics=<factory>)[source]¶

Bases: object

Aggregated cross-validation output for one exported model.

Parameters:

export_dir (str) – Path to the exported model bundle directory.
task (str) – Task type (e.g. regression or screening).
n_folds (int) – Requested number of folds.
effective_folds (int) – Folds actually evaluated after grouping constraints.
strategy (str) – Split strategy used (row_kfold or receptor_grouped).
epochs (int) – Training epochs per fold.
random_seed (int) – Random seed used for splits and training.
objective_metric (str) – Primary metric optimized during training.
fold_results (list of CrossValidationFoldResult) – Per-fold metrics and diagnostics.
aggregate_validation_metrics (dict) – Mean/std (or similar) aggregates across folds for the primary model.
model_config (dict) – Serialized model configuration from the export bundle.
scoring_function_columns (list of str, optional) – Scoring-function columns evaluated as baselines, by default empty list.
aggregate_scoring_function_metrics (dict, optional) – Aggregated baseline metrics by scorer, by default empty dict.
scorer_comparison_summary (dict, optional) – Summary comparing OCScore vs baselines, by default empty dict.
diagnostics (dict, optional) – Run-level diagnostics, by default empty dict.

export_dir: str¶

task: str¶

n_folds: int¶

effective_folds: int¶

strategy: str¶

epochs: int¶

random_seed: int¶

objective_metric: str¶

fold_results: list[CrossValidationFoldResult]¶

aggregate_validation_metrics: dict[str, dict[str, float]]¶

model_config: dict[str, Any]¶

scoring_function_columns: list[str]¶

aggregate_scoring_function_metrics: dict[str, dict[str, dict[str, float]]]¶

scorer_comparison_summary: dict[str, Any]¶

diagnostics: dict[str, Any]¶

OCDocker.OCScore.Optimization.ModelCrossValidation.build_scorer_comparison_summary(result, *, comparison_metrics=None, reference_scorer='OCScore')[source]¶

Summarize OCScore vs scoring-function baselines across CV folds.

Computes mean ± std per scorer/metric, per-fold rankings (1 = best), and how often reference_scorer achieves the top value on each fold.

Parameters:

result (CrossValidationResult) – Completed cross-validation output.
comparison_metrics (Sequence[str] | None, optional) – Metrics to summarize. Defaults to DUDEZ_CV_COMPARISON_METRICS for DUDEz screening and PDBBIND_CV_METRICS for PDBbind regression.
reference_scorer (str, optional) – Scorer used for win counting, by default OCScore.

Returns:

Summary tables: mean_std, ocscore_wins, fold_rankings.

Return type:

dict[str, Any]

OCDocker.OCScore.Optimization.ModelCrossValidation.diagnose_entity_overlap(dataframe, train_idx, val_idx, entity_columns)[source]¶

Report duplicate entity keys between train and validation splits.

Parameters:

dataframe (DataFrame)
train_idx (ndarray)
val_idx (ndarray)
entity_columns (Sequence[str])

Return type:

dict[str, Any]

OCDocker.OCScore.Optimization.ModelCrossValidation.evaluate_scoring_function_baselines_on_fold(dataframe, validation_indices, labels, groups, scoring_columns, *, metric_names=('BEDROC', 'ROC-AUC', 'PR-AUC', 'EF1%', 'EF5%', 'NDCG@1%', 'NDCG@5%', 'Precision', 'Recall', 'F1', 'MCC', 'TP', 'FP', 'TN', 'FN'), bedroc_alpha=20.0)[source]¶

Evaluate individual scoring functions on one CV validation fold.

Parameters:

dataframe (pd.DataFrame) – Full reduced DUDEz dataframe.
validation_indices (np.ndarray) – Row indices for the validation fold.
labels (np.ndarray) – Binary active/decoy labels aligned with dataframe rows.
groups (np.ndarray | None) – Receptor groups for grouped screening metrics.
scoring_columns (Sequence[str]) – Scoring-function columns to evaluate.
metric_names (Sequence[str], optional) – Metrics retained in each returned scorer dictionary.
bedroc_alpha (float, optional) – BEDROC alpha used for scorer baseline BEDROC, by default 20.0.

Returns:

Mapping from scorer name to validation metrics on this fold.

Return type:

dict[str, dict[str, float]]

OCDocker.OCScore.Optimization.ModelCrossValidation.identify_scoring_function_columns(selected_features)[source]¶

Return scoring-function descriptor columns from the selected feature list.

Parameters:: selected_features (Sequence[str])
Return type:: list[str]

OCDocker.OCScore.Optimization.ModelCrossValidation.infer_higher_is_better(scores, labels)[source]¶

Infer whether larger raw scores favor actives on one validation fold.

Parameters:

scores (ndarray)
labels (ndarray)

Return type:

bool

OCDocker.OCScore.Optimization.ModelCrossValidation.iter_receptor_group_kfold_indices(groups, n_folds, *, random_seed, shuffle)[source]¶

Return train/validation index pairs by holding out whole receptor groups.

Parameters:

groups (ndarray)
n_folds (int)
random_seed (int)
shuffle (bool)

Return type:

list[tuple[ndarray, ndarray]]

OCDocker.OCScore.Optimization.ModelCrossValidation.iter_row_kfold_indices(n_samples, n_folds, *, random_seed, shuffle)[source]¶

Return train/validation index pairs using row-wise K-fold.

Parameters:

n_samples (int)
n_folds (int)
random_seed (int)
shuffle (bool)

Return type:

list[tuple[ndarray, ndarray]]

OCDocker.OCScore.Optimization.ModelCrossValidation.run_cross_validation_from_export(export_dir, dataframe, *, config=None, device=None, output_dir=None, transferred_extractor=None)[source]¶

Run K-fold cross-validation using a fixed exported model configuration.

Parameters:

export_dir (str | Path) – Exported best_model/ directory.
dataframe (pd.DataFrame) – Reduced PDBbind or DUDEz dataframe aligned with the export features.
config (CrossValidationConfig | None, optional) – Cross-validation settings including n_folds.
device (torch.device | str | None, optional) – Training device, by default CPU.
output_dir (str | Path | None, optional) – Directory for cross_validation_results.json and fold CSV. Defaults to <export_dir>/cross_validation.
transferred_extractor (FeatureExtractor | None, optional) – Optional transferred extractor for DUDEz transfer exports.

Returns:

Per-fold and aggregate validation metrics.

Return type:

CrossValidationResult

OCDocker.OCScore.Optimization.ModelCrossValidation.save_cross_validation_result(result, output_dir)[source]¶

Write cross-validation JSON and per-fold CSV artifacts.

Parameters:

result (CrossValidationResult) – Cross-validation output.
output_dir (str | Path) – Destination directory.

Returns:

Written artifact paths.

Return type:

dict[str, str]

OCDocker.OCScore.Optimization.ModelCrossValidation.validate_fold_indices(train_idx, val_idx, *, fold_index)[source]¶

Assert train and validation row indices are disjoint.

Parameters:

train_idx (ndarray)
val_idx (ndarray)
fold_index (int)

Return type:

None

OCDocker.OCScore.Optimization.ModelCrossValidation.validate_fold_split(train_idx, val_idx, *, fold_index, strategy, groups=None)[source]¶

Run fold integrity checks before training.

Parameters:

train_idx (ndarray)
val_idx (ndarray)
fold_index (int)
strategy (str)
groups (ndarray | None)

Return type:

None

OCDocker.OCScore.Optimization.ModelCrossValidation.validate_receptor_group_split(groups, train_idx, val_idx, *, fold_index)[source]¶

Assert held-out receptors do not appear in training rows.

Parameters:

groups (ndarray)
train_idx (ndarray)
val_idx (ndarray)
fold_index (int)

Return type:

None