Skip to content

Add shuffle kwarg to GroupBy.map - #11879

Open
PStarH wants to merge 2 commits into
pydata:mainfrom
PStarH:groupby-map-shuffle
Open

PStarH wants to merge 2 commits into
pydata:mainfrom
PStarH:groupby-map-shuffle

Conversation

@PStarH

@PStarH PStarH commented Nov 13, 2026

Copy link
Copy Markdown

Problem

GroupBy.map builds one task and one graph layer per group. For dask-backed data with many groups this dominates both graph construction and scheduling; #9220 shows the effect for out-of-order labels such as time.dayofyear:

100×100×4017, chunks (100, 100, 365), map(lambda g: g - g.mean("time")):

output chunks tasks layers graph build
map(...) 402 12,494 2,199 0.700 s
map(..., shuffle=True) 12 486 7 0.020 s

Approach

Add a keyword-only shuffle: bool = False to DataArrayGroupBy.map and DatasetGroupBy.map:

ds.groupby("time.dayofyear").map(center, shuffle=True)

When the object is chunked and shuffle=True:

  1. shuffle the object with the existing _shuffle_obj, so every group is contained in one chunk;
  2. apply func once per block with map_blocks, using an inner groupby(...).map;
  3. restore the original order with _inverse_permutation_indices + _shuffle.

A cheap template is inferred by applying func to one group and reusing the shuffled object's metadata (dtypes only); functions that change dims/variables fall back to full per-group inference. Reductions are reindexed to the full group index, so empty groups (bins/resample-style) match the baseline, including NaNs.

The default (shuffle=False) and non-chunked inputs use the existing code path unchanged. New tests compare against the baseline with assert_identical.

This builds on the approach explored in the draft #9706, updated to current main with tests, docs and the empty-group reduction fix.

Benchmarks

Compute on 55×55×2191 (≈53 MB), scheduler="threads", graph build + compute per run, median of 3: 1.317 s → 0.296 s (4.45×; baseline runs 1.372 / 1.317 / 1.280 s, shuffle runs 0.298 / 0.272 / 0.296 s).

Limitations

  • func must preserve the size of the grouped dimension; shrinking/growing functions raise NotImplementedError when shuffle=True.
  • Chunked groupers, multi-dimensional groupers, multiple groupers, and resampling are not supported with shuffle=True (chunked groupers are also unsupported by map today).
  • xarray objects in args and functions that change coordinate values are not covered by the template fast path yet.
  • shortcut is accepted but has no effect in the shuffled path.

Tests

15 new tests cover element-preserving DataArray/Dataset, mixed dtypes, derived and out-of-order groupers, reductions (including empty groups), NaN groups, bins, args/kwargs, non-chunked fallback, graph size, and raise_if_dask_computes.

Regression sweep (test_groupby + test_dask + test_rolling + test_cftimeindex_resample): 4,006 passed, 773 skipped, 3 xfailed. Full suite: 18,365 passed, 3,634 skipped, 65 xfailed, 24 xpassed (337 s, exit 0).

This PR was prepared with AI assistance (Claude); all reported numbers were measured on the patched tree, and every supported case is checked against the baseline with assert_identical.

🤖 Generated with Claude Code

@welcome

welcome Bot commented Nov 13, 2026

Copy link
Copy Markdown

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

Shuffle the object so each group is contained in one chunk, apply the
function once per block with map_blocks, then restore the original order.
For out-of-order group labels this reduces the dask graph from O(groups)
tasks to O(chunks), with a template fast path and full-inference fallback.

Closes pydata#9220. Builds on the approach explored in pydata#9706.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@PStarH
PStarH force-pushed the groupby-map-shuffle branch from b716fb9 to 3c0e6e5 Compare November 13, 2026 03:19
Counting group occurrences with a groupby reduction may dispatch to
flox/numbagg, which raises a numba TypingError for object-like group
labels such as intervals. Use pandas value_counts instead; the result is
reindexed to the full group index exactly as before.

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant