[qbarthelemy]

Solve #1221 (as a tutorial, after review)

Cross-system transfer, where each system is an EEG device with its own montage and sampling rate, is handled by the existing API instead of a new evaluation class:

• the devices' datasets are pooled in a CompoundDataset;
• the paradigm keeps every trial with its device's channels and sampling rate (nested trials, Let get_data return nested trials when recordings differ in channels or samples #1327);
• CrossSubjectEvaluation(cv_class=LeaveOneGroupOut, groups="device") holds out one device at a time, using the device metadata column (Add dataset and device columns to the paradigm metadata (leave-one-device-out with groups="device") #1328). A device is not a dataset: one dataset can use several devices, several datasets can share one.

This PR now contains

• the tutorial examples/how_to_benchmark/plot_cross_system_transfer_maskedmean.py: union of channels, mask matrices, minimum distance to masked Riemannian mean vs. zero-filled and channel-intersection MDM. The pipeline steps read the device of each training trial and infer the held-out device, so every device is the target in turn;
• generic metadata routing in _evaluate_fold: any metadata column of the training trials (device, dataset, subject...) is passed to the steps that request it with set_fit_request(<column>=True), only when requested;
• a device argument for FakeDataset;
• a fix of process() returning results several times when datasets share a processing pipeline (found here, with its test).

CrossSystemEvaluation and its cross-system-only routing are removed.

Tests: metadata-column routing (requested / not requested), leave-one-device-out routing only the training devices, duplicated results; the tutorial runs end to end. test_evaluations.py, test_paradigms.py and test_splits.py pass locally.

@bruAristimunha @sylvchev @plcrodrigues

[bruAristimunha]

Thanks @qbarthelemy, the evaluation fits #1221 well, and the metadata routing of systems and the per-dataset result filtering are clean. A review round before it can go in (the branch also needs develop merged; it conflicts now):

Should fix

1. __init__: self.datasets = list(target_datasets) replaces the list after BaseEvaluation.__init__ has filtered it, so targets skip the paradigm check, and a target that is not in datasets produces no fold and returns an empty result without any message. Suggest validating target_datasets against the filtered self.systems (same code) and raising otherwise.
2. _load_data loads every system in full on each call, ignoring subjects, and is called once per target: leave-one-system-out on k systems does k full loads of all k datasets. Loading the systems once per process() (e.g. keep the concatenated X, y, metadata on the instance for the duration of the run) avoids the k² reads.
3. _load_data also ignores run_pipes / process_pipeline: a pipeline that needs Epochs (TRCA, braindecode, ...) silently receives arrays, and a filter-bank paradigm gets a single band. Raising a clear ValueError for epoch-requiring pipelines (as done for postprocess_pipeline) is enough for now.

Minor

4. With datasets=None, every dataset of the paradigm becomes a system and is loaded for each target; worth requiring explicit datasets here.
5. The _process_parallel change (keep only the current dataset's rows) also fixes duplicated rows for the other evaluations when datasets share a pipeline digest; it deserves its own Bugs entry in whats_new.rst.

Happy to help with the develop merge once these are settled.

[bruAristimunha]

@qbarthelemy I pushed two commits to your branch so this can go in: a merge of develop (keep-both on whats_new.rst and the tests) and 9f04831, the smallest fix for review points 1-3: the systems are loaded once per evaluation instead of once per target, a target_datasets entry outside datasets raises, and pipelines that need mne.Epochs raise instead of silently getting arrays (one new test). Points 4-5 stay optional. Shout if you'd rather do it differently.

[bruAristimunha]

@qbarthelemy following the review here, we went the other way round: instead of a new evaluation, the existing API now covers cross-system transfer, and your example becomes a tutorial.

• Let get_data return nested trials when recordings differ in channels or samples #1327 (merged): get_data returns nested trials (object array of per-trial (n_channels, n_times) arrays) when recordings differ in channels or sampling rate, and the evaluations accept them.
• Add dataset and device columns to the paradigm metadata (leave-one-device-out with groups="device") #1328 (in CI): the paradigm metadata gets dataset and device columns. A dataset is not a device (one dataset can use several, several datasets can share one), so cross-system transfer is CrossSubjectEvaluation(datasets=[CompoundDataset(...)], cv_class=LeaveOneGroupOut, groups="device").

Once #1328 is in, I'd like to turn this PR into the tutorial: keep your example (union of channels, masks, masked-mean classifier vs. channel intersection), run it through CompoundDataset + groups="device", and drop CrossSystemEvaluation and its evaluation changes. Authorship stays yours. If you'd rather do it yourself or keep a different shape, say so before I push.

[bruAristimunha]

@qbarthelemy as announced above (no objection after a day), I pushed bf93c3d to turn this into the tutorial: CrossSystemEvaluation is gone, your example runs on CompoundDataset + CrossSubjectEvaluation(cv_class=LeaveOneGroupOut, groups="device"), and your pipeline steps request the device metadata column (any column can now be requested that way) instead of systems. Your process() duplicate-results fix stays, credited to you in the changelog. Description updated. If you'd rather adjust anything, push on top; nothing was force-pushed.

[qbarthelemy]

Sorry, I’m often AFK on the weekends.
The way you’ve reworked the PR is great—thanks!