[k82cn]

TestGangScheduling::test_single_instance_sessions_are_not_held_back fails intermittently in the Python E2E jobs (seen on #621, #624, drf and priority). The cluster state printed by #630 showed why: six idle executors left by the previous test, all Bound to the 7-instance gang at the timeout, and none for the single session.

Cause

_create_gang creates the session with min_instances=size and batch_size=size, then its task. batch_size() in scheduler/plugins/mod.rs counted the gang only while the session had a pending task, so between the two calls the session was not a gang: min_instances made it underused, is_satisfied was trivially true, and Dispatch bound idle executors to it one by one. A gang of 7 on a cluster of 6 then held all six, could not get a seventh, and was not over its demand, so Shuffle did not take them back; the single session never got an executor. A scheduler cycle (every 100 ms) has to land in the few milliseconds between the two calls, which is why it is intermittent. Reproduced deterministically in the in-process scheduler with a cycle between the two calls; the new scheduler test below is that scenario.

Change

A gang (batch_size above 1) is the whole session, so its instance limits are tied to its size:

• Validation. create_session, and open_session with a spec, reject (INVALID_ARGUMENT) a session with batch_size above 1 unless min_instances is 0 or batch_size and max_instances is unset or batch_size, and unless the application's max_instances is at least batch_size. A session without a gang is rejected only if its max_instances is below its min_instances; a max_instances of 0 without a gang stays valid (it prevents allocation, as documented).
• Scheduler. MinMax's is_satisfied treats a gang kept warm (batch_size above 1 and min_instances above 0, which the validation makes equal) as a gang whether or not a task is pending, so it is formed all at once like any gang, even before its first task. The demand already includes min_instances, so Shuffle does not reclaim its executors (is_overused is false at batch_size bound). batch_size() in plugins/mod.rs is unchanged.

Nothing else in Dispatch, Allocate or Shuffle changes. The e2e gang tests are unchanged: they already create gangs with min_instances=size.

Formatting

rustfmt.toml sets single_line_if_else_max_width = 60 (the default is 50), so a short if/else such as let kept_warm = if ssn.batch_size > 1 { ssn.min_instances } else { 0 }; stays on one line. cargo fmt --all reformatted three other if/else expressions to match (common/src/storage/engine/sqlite.rs, executor_manager/src/appmgr/downloader.rs, flmctl/src/list.rs); the changes are whitespace only. A branch that is not rebased will fail cargo fmt --check on those spots until it runs cargo fmt.

Compatibility

Sessions with batch_size above 1 and another min_instances or max_instances are now rejected when created. In the repository, only examples/rl/search-r1's trainer sets a gang (batch_size=TRAINERS, no warmup, autoscaling: min_instances 0 and max_instances unset), which stays valid. From the Python SDK, warmup must be 0 or batch_size, and autoscale=False (which fixes the maximum at warmup) needs warmup=batch_size. RFE591, docs/api/types.md and sdk/python/docs/API.md say so.

Tests

• controller/mod.rs: gang_min_and_max_instances_are_its_size_or_not_set: each rejected combination through create_session and open_session, and the accepted ones (gang alone, gang kept warm, exact limits, no gang with its own limits, a limit of 0 without a gang).
• scheduler/mod.rs: test_gang_kept_warm_binds_all_at_once_even_without_a_task, with priority+das and drf+das: a gang of 7 with min_instances 7 on 6 slots binds nothing with or without a task, binds all 7 on 7 slots with or without a task, and one with min_instances 0 and no task binds nothing. It fails with main's batch_size().
• plugins/minmax.rs: a gang kept warm is satisfied only when whole before its first task, and is never over its demand; the demand tests cover min_instances equal to batch_size.
• cargo test -p flame-session-manager (299 passed with --test-threads=6; at the default parallelism some tests hit the per-process file limit on my machine), clippy -D warnings, rustfmt.
• Not run: the gang e2e tests (they need a cluster; a race is not proved fixed by one green run, the scheduler test above is the evidence) and the Python and Rust SDK suites.

This replaces the earlier contents of this pull request (a different limit rule and a rework of the e2e gang tests) and supersedes #637.