Skip to content

fix(models): encode labels for probability metrics in evaluate and score - #494

Merged
mkumar73 merged 3 commits into
mainfrom
fix/evaluate-encoded-labels
Oct 31, 2026
Merged

mkumar73 merged 3 commits into
mainfrom
fix/evaluate-encoded-labels

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #493

Problem

SklearnBaseClassifier.evaluate() passed the original y_true labels to the probability-based metrics, together with predict_proba() output whose columns follow classes_. score(X, y, metric=(metric, True)) did the same. Two failures followed:

  • For string labels, AUPRC, BrierScore and ExpectedCalibrationError failed internally and were silently reported as nan.
  • For integer labels other than 0..K-1 (e.g. {1, 2}), AUPRC scored the wrong class (0.302 instead of 1.0) and ECE became nan.

Fix

  • A new helper, _probability_metric_targets, encodes y_true with the existing _encode_labels(…, classes_). That is the same 0..K-1 mapping fit() and predict() use, and it matches the predict_proba() column order.
  • evaluate() passes the encoded labels to the probability-based metrics (auroc, auprc, log_loss, brier, ece). Label metrics (accuracy, F1, …) still receive the original labels next to the decoded predict() output.
  • score(X, y, metric=(metric, True)) does the same for those DeepTab probability metrics. Other callables keep receiving the original labels, so user-supplied functions see the labels they were given.
  • Labels that cannot be encoded (a label never seen during fit, or a None) fall back to the raw labels. The existing per-metric error handling then applies, so evaluate() itself doesn't start raising where it didn't before.

The names of the probability metrics move from a local set in evaluate() to the module-level _PROBA_NAMES, so both methods use the same list.

#518 (LogLoss with missing classes) treats valid integer labels as column indices and relies on evaluate()/score() passing encoded labels, so this PR should be merged before it.

Tests

tests/integration/test_label_encoding.py::TestClassifierLabelEncoding:

  • test_probability_metrics_receive_encoded_labels, for string binary, {1, 2}, {10, 20} and string multiclass labels: every probability metric is finite, and both evaluate() and score(..., metric=(metric, True)) equal the same metric computed directly on the encoded labels and predict_proba(). Binary-only metrics are used only in the binary cases.
  • test_evaluate_reports_nan_for_labels_that_cannot_be_encoded: a None in y_true gives nan, as on main, instead of a TypeError.

The 4 parametrized cases fail on main and pass with this change. The None case guards the fallback.

Docs

The evaluate() and score() numpydoc say that DeepTab's probability metrics receive labels encoded to the classes_ column indices, and everything else the original labels.

Checks

  • just test: 3241 passed, 46 skipped, 28 xfailed; coverage 91.37%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

evaluate() passed the raw y_true labels to AUROC, AUPRC, LogLoss,
BrierScore and ExpectedCalibrationError next to the predict_proba()
columns, which follow classes_. With string labels AUPRC, Brier and ECE
silently returned NaN, and with integer labels other than 0..K-1 AUPRC
scored the wrong class. Probability metrics now receive y_true encoded
to the column indices of classes_, while label metrics keep the original
labels.

Fixes #493
score(X, y, metric=(metric, True)) passed the raw labels to DeepTab's
probability metrics, like evaluate() did. Labels that cannot be encoded
(such as None) no longer make evaluate() raise a TypeError.

Refs #493
@ChrisW09 ChrisW09 changed the title fix(models): encode labels for probability metrics in evaluate fix(models): encode labels for probability metrics in evaluate and score Oct 21, 2026
@mkumar73
mkumar73 merged commit 0b6aed1 into main Oct 31, 2026
35 checks passed
@mkumar73
mkumar73 deleted the fix/evaluate-encoded-labels branch October 31, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Classifier evaluate() and score() pass raw labels to probability metrics: AUPRC/Brier/ECE are NaN or wrong for string or non-0..K-1 labels

2 participants