Skip to content

fix(distributions): make distribution families picklable - #558

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/picklable-distributions
Open

ChrisW09 wants to merge 1 commit into
mainfrom
fix/picklable-distributions

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #557

Problem

BaseDistribution.__init__ stored the "none", "square", "probabilities" and "log" parameter transforms as lambdas, and get_transform() fell back to another lambda. Lambdas defined inside a function can't be pickled, so pickling failed with Can't get local object 'BaseDistribution.__init__.<locals>.<lambda>' for:

  • every distribution family;
  • every fitted LSS estimator;
  • anything that pickles the LSS training module, such as Lightning's ddp_spawn strategy.

Fix

deeptab/distributions/base.py replaces the lambdas with module-level functions (_identity, _square, _softmax_last_dim, _stable_log) that compute exactly the same values. The get_transform() fallback uses _identity. The behaviour is unchanged otherwise. Unknown transform names still map to the identity; that's a separate question.

Tests

In tests/integration/test_save_load.py, the module that already holds the pickle tests and has the S301 ruff exemption:

  • test_every_distribution_family_survives_a_pickle_round_trip[<all 16 registered families>]: same class and parameter names after pickle.loads(pickle.dumps(...)). For normal, the restored family also produces the same transformed parameters.
  • test_fitted_lss_can_be_pickled[normal|quantile|categorical]: a fitted MLPLSS pickles and unpickles as an unfitted estimator, as documented for the other estimators.

All 19 fail on main and pass with this change. I confirmed the ddp_spawn failure on main (MLPLSS(...).fit(..., accelerator="cpu", devices=2, strategy="ddp_spawn") raises the same AttributeError). In my sandbox, an end-to-end ddp_spawn run can't complete for any estimator, including MLPRegressor, because torch_shm_manager can't bind. With this change the LSS run gets past pickling to that same environment error.

Docs

No change needed.

Checks

  • just test: 3255 passed, 46 skipped, 28 xfailed; coverage 91.28%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

The parameter transforms of BaseDistribution were lambdas, so pickling a
distribution family, a fitted LSS estimator or the LSS training module
(e.g. for Lightning's ddp_spawn strategy) failed with "Can't get local
object". Use module-level functions with the same behaviour.

Fixes #557

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Distribution families and fitted LSS estimators cannot be pickled (local lambdas in BaseDistribution)

1 participant