[BUG] LogLoss fails when y_true lacks a class: validation metric crashes fit(), evaluate() reports NaN #517

Describe the bug

LogLoss fails whenever y_true doesn't contain every class. Small and imbalanced validation or test sets often lack a class, and LogLoss is a default classification metric, so this matters in practice:

• As a training metric (val_metrics={"log_loss": LogLoss()}), it raises ValueError and stops fit() when the validation set lacks a class.
• evaluate() with the default classification metrics (Accuracy, AUROC, LogLoss) catches the error and reports log_loss = nan.

LogLoss calls sklearn.metrics.log_loss(y_true, y_pred) without labels. scikit-learn then infers the classes from y_true and refuses a probability matrix with more columns than observed classes. DeepTab passes probability columns ordered by the encoded classes 0..K-1, so those column indices are the class labels.

To Reproduce

import numpy as np, pandas as pd
from deeptab.configs import TrainerConfig
from deeptab.metrics import LogLoss
from deeptab.models import MLPClassifier

probabilities = np.array([[0.7, 0.2, 0.1], [0.1, 0.8, 0.1], [0.6, 0.3, 0.1], [0.2, 0.7, 0.1]])
try:
    print("1) LogLoss on labels {0, 1} with 3 classes:", LogLoss()(np.array([0, 1, 0, 1]), probabilities))
except Exception as exc:
    print(f"1) LogLoss on labels {{0, 1}} with 3 classes: {type(exc).__name__}: {exc}")

rng = np.random.default_rng(0)
X = pd.DataFrame({"a": rng.normal(size=90), "b": rng.normal(size=90)})
y = np.tile([0, 1, 2], 30)
keep = y != 2  # a validation / test set in which class 2 doesn't occur
fit_kwargs = dict(accelerator="cpu", logger=False, enable_progress_bar=False, enable_model_summary=False)
model = MLPClassifier(trainer_config=TrainerConfig(max_epochs=1, batch_size=16), random_state=0)
try:
    model.fit(X[:60], y[:60], X_val=X[60:][keep[60:]], y_val=y[60:][keep[60:]], val_metrics={"log_loss": LogLoss()}, **fit_kwargs)
    print("2) fit with val_metrics={'log_loss': LogLoss()}: ok")
except Exception as exc:
    print(f"2) fit with val_metrics={{'log_loss': LogLoss()}}: {type(exc).__name__}: {exc}")
model = MLPClassifier(trainer_config=TrainerConfig(max_epochs=1, batch_size=16), random_state=0).fit(X[:60], y[:60], **fit_kwargs)
print("3) evaluate() log_loss on data without class 2:", model.evaluate(X[keep], y[keep])["log_loss"])

Output:

1) LogLoss on labels {0, 1} with 3 classes: ValueError: y_true and y_pred contain different number of classes 2, 3. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1]
2) fit with val_metrics={'log_loss': LogLoss()}: ValueError: y_true and y_pred contain different number of classes 2, 3. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1]
3) evaluate() log_loss on data without class 2: nan

Expected behavior

For a (N, K) probability matrix and integer class indices 0..K-1, the classes are the matrix columns, and the log loss is defined whatever subset of them occurs in y_true. Integer labels that can't be column indices (e.g. raw labels {1, 2, 3} with three columns) keep scikit-learn's own class inference.

Note that evaluate() and score(..., metric=(LogLoss(), True)) currently pass raw labels to probability metrics (#493, fixed by #494). Raw integer labels that happen to fall inside 0..K-1 but are not the encoded indices (e.g. {1, 2} from classes {1, 2, 3}) can only be scored correctly once those callers encode the labels, so #494 should be merged first.

Where

deeptab/metrics/classification.py:206-207 (LogLoss.__call__): _log_loss(y_true, y_pred) is called without labels.

Environment

• OS: macOS 26.6 (arm64)
• Python: 3.12.13
• deeptab: 2.1.0 (main @ ca18a4e), scikit-learn 1.6.1, torch 2.14.1