Repository navigation
fix(models): encode labels for probability metrics in evaluate and score - #494
Merged
Merged
Conversation
evaluate() passed the raw y_true labels to AUROC, AUPRC, LogLoss, BrierScore and ExpectedCalibrationError next to the predict_proba() columns, which follow classes_. With string labels AUPRC, Brier and ECE silently returned NaN, and with integer labels other than 0..K-1 AUPRC scored the wrong class. Probability metrics now receive y_true encoded to the column indices of classes_, while label metrics keep the original labels. Fixes #493
score(X, y, metric=(metric, True)) passed the raw labels to DeepTab's probability metrics, like evaluate() did. Labels that cannot be encoded (such as None) no longer make evaluate() raise a TypeError. Refs #493
mkumar73
approved these changes
Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #493
Problem
SklearnBaseClassifier.evaluate()passed the originaly_truelabels to the probability-based metrics, together withpredict_proba()output whose columns followclasses_.score(X, y, metric=(metric, True))did the same. Two failures followed:AUPRC,BrierScoreandExpectedCalibrationErrorfailed internally and were silently reported asnan.0..K-1(e.g.{1, 2}),AUPRCscored the wrong class (0.302 instead of 1.0) and ECE becamenan.Fix
_probability_metric_targets, encodesy_truewith the existing_encode_labels(…, classes_). That is the same0..K-1mappingfit()andpredict()use, and it matches thepredict_proba()column order.evaluate()passes the encoded labels to the probability-based metrics (auroc,auprc,log_loss,brier,ece). Label metrics (accuracy, F1, …) still receive the original labels next to the decodedpredict()output.score(X, y, metric=(metric, True))does the same for those DeepTab probability metrics. Other callables keep receiving the original labels, so user-supplied functions see the labels they were given.fit, or aNone) fall back to the raw labels. The existing per-metric error handling then applies, soevaluate()itself doesn't start raising where it didn't before.The names of the probability metrics move from a local set in
evaluate()to the module-level_PROBA_NAMES, so both methods use the same list.#518 (
LogLosswith missing classes) treats valid integer labels as column indices and relies onevaluate()/score()passing encoded labels, so this PR should be merged before it.Tests
tests/integration/test_label_encoding.py::TestClassifierLabelEncoding:test_probability_metrics_receive_encoded_labels, for string binary,{1, 2},{10, 20}and string multiclass labels: every probability metric is finite, and bothevaluate()andscore(..., metric=(metric, True))equal the same metric computed directly on the encoded labels andpredict_proba(). Binary-only metrics are used only in the binary cases.test_evaluate_reports_nan_for_labels_that_cannot_be_encoded: aNoneiny_truegivesnan, as onmain, instead of aTypeError.The 4 parametrized cases fail on
mainand pass with this change. TheNonecase guards the fallback.Docs
The
evaluate()andscore()numpydoc say that DeepTab's probability metrics receive labels encoded to theclasses_column indices, and everything else the original labels.Checks
just test: 3241 passed, 46 skipped, 28 xfailed; coverage 91.37%just docs: build succeeded (warnings treated as errors)just check: all hooks passed (ruff, prettier, pyright)