Skip to content

fix(metrics): pass the probability columns as labels in LogLoss - #518

Merged
mkumar73 merged 5 commits into
mainfrom
fix/logloss-missing-classes
Oct 31, 2026
Merged

mkumar73 merged 5 commits into
mainfrom
fix/logloss-missing-classes

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #517

Problem

LogLoss called sklearn.metrics.log_loss(y_true, y_pred) without labels. scikit-learn then infers the classes from y_true and raises when the probability matrix has more columns than observed classes. Any validation or test set that lacks a class therefore:

  • stopped fit() when LogLoss was a training/validation metric;
  • produced log_loss = nan in evaluate(), where LogLoss is one of the default classification metrics.

Fix

For a 2-D probability matrix with more than one column and integer labels that are all valid column indices (0 <= y < K), LogLoss passes labels=np.arange(K). The probability columns are ordered by the encoded classes 0..K-1, as in the training loop and predict_proba(). Other inputs (1-D positive-class probabilities, non-integer labels, and integers outside 0..K-1 such as raw labels {1, 2, 3}) are passed to scikit-learn unchanged, which infers the classes from y_true as before. The class docstring documents this.

Merge after #494. evaluate() and score(..., metric=(LogLoss(), True)) pass raw labels to probability metrics on main (#493). Raw integer labels that lie inside 0..K-1 without being the encoded indices (e.g. {1, 2} from classes {1, 2, 3}) can't be told apart from encoded ones here; with #494 these callers pass encoded labels, so the scores are correct in every case.

Tests

tests/metrics/test_classification_metrics.py::TestLogLossWithMissingClasses:

  • test_multiclass_matrix_uses_all_columns_as_classes
  • test_binary_matrix_with_a_single_observed_class
  • test_evaluate_and_validation_metrics_on_data_without_one_class: an MLPClassifier with val_metrics={"log_loss": LogLoss()} and a validation set lacking class 2 fits, and evaluate() on data without class 2 returns a finite log loss.
  • test_labels_outside_the_column_indices_are_inferred_by_sklearn: LogLoss()([1, 2, 3], P) equals log_loss([1, 2, 3], P).
  • test_evaluate_with_labels_other_than_column_indices: for an MLPClassifier trained on labels {1, 2, 3}, evaluate() equals log_loss(y, predict_proba(X), labels=classes_).

The first three fail on main and pass with this change. The last two pass on main too; they guard against scoring raw labels as column indices.

Docs

The LogLoss docstring, shown in the API reference, describes the behaviour. No page needed a change.

Checks

  • just test: 3241 passed, 46 skipped, 28 xfailed; coverage 91.32%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

sklearn's log_loss infers the classes from y_true, so LogLoss raised for
any evaluation set that lacks a class: as a validation metric it stopped
fit(), and evaluate() reported NaN. For a 2-D probability matrix and
integer class indices, pass labels=0..K-1 so the score is defined for
any subset of observed classes.

Fixes #517
Integer labels outside 0..K-1 (for example classes 1, 2, 3) are raw
class labels, not column indices. Passing labels=0..K-1 for them made
scikit-learn score the wrong columns without a warning, so they are
left to scikit-learn's class inference again.

Refs #517
@mkumar73
mkumar73 merged commit 01095a5 into main Oct 31, 2026
35 checks passed
@mkumar73
mkumar73 deleted the fix/logloss-missing-classes branch October 31, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] LogLoss fails when y_true lacks a class: validation metric crashes fit(), evaluate() reports NaN

2 participants