Skip to content

feat(AUTO-1062): ask sandbox for variable mutations only and rebuild scopes on the host - #1568

Merged
Kart03 merged 7 commits into
developfrom
feat/mutations-only-result
Oct 22, 2026
Merged

Kart03 merged 7 commits into
developfrom
feat/mutations-only-result

Conversation

@Kart03

@Kart03 Kart03 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Context

AUTO-1062 / AUTO-60. After each script the sandbox sends back the full local, environment, globals and collectionVariables scopes, and runtime mostly needs only their mutations. With ~4 MB of variables that copy costs ~22 ms and ~22 MB per script. One sandbox thread serves all VUs, so it also caps throughput: in a 44-VU, 20-minute CLI run, a prototype cut the worker peak from 2.65 to 1.92 GB and raised throughput from 32 to 40 rps.

Summary of the change

  • New run option script.variableMutationsOnly (default false). When set, event.command.js passes variableMutationsOnly: true to host.execute (needs feat(AUTO-1062): add variableMutationsOnly option to return only scope mutations postman-sandbox#1116). Consumers that don't set it keep today's full-scope round trip. The perf lib opts in.
  • When the option is set and a result scope (_variables, environment, globals, collectionVariables) comes back with no values, runtime rebuilds it with applyScopeMutations: a copy of contextToUse[scope] with the mutations applied, and .mutations kept. The script trigger still gets full scopes with mutations, which nested-request and consumers read.
  • contextToUse is the masked clone when _forbiddenSecretKeys apply, so masked secrets stay masked in the script result. A sandbox-only prototype failed 9 tests here (8 vault masking, 1 pm.variables).
  • Tracked scopes are still updated through result[scope].mutations → applyOn(payload.context[scope]), as before.
  • The rebuild copies Variable objects but shares their string values, so it stays cheap even with large values.

Breaking changes

None. The option is off by default, so existing consumers see no change. With it on, an older sandbox ignores it and sends full scopes, which take the old path (detected by values).

Dependencies

postmanlabs/postman-sandbox#1116 (variableMutationsOnly). No gain until runtime bumps postman-sandbox. Until then this PR is behaviour-neutral.

Test plan

Suite sandbox 6.7.4 sandbox branch
test-unit 571 passing 571 passing
test-integration 1901 passing, 0 failing 1901 passing, 0 failing
test-integration-legacy not run 33 passing
test-lint ok ok

Bench: node copybench.js <runtimeDir> 7 100 (1 request, prerequest + test script, 4.08 MB collection vars, 200 scripts):

ms/script host alloc MB/script sandbox→host MB/script
develop 22.8 / 22.7 / 25.3 22.4 / 22.5 / 22.9 4.55
this + sandbox branch 13.4 / 13.6 / 13.8 14.2 / 14.1 / 14.1 0

Opt-in commit (a2c60f2): new test/integration/sanity/variable-mutations-only.test.js covers default off, opt-in, and the host rebuild when the sandbox sends mutations only (stubbed, so it runs against any sandbox version). Its default-off case fails against the previous hardcoded true. test-integration: 1918 passing, 0 failing. test-unit: 571 passing. The one failure, version.test.js, is down to local node_modules not matching the lockfile, not this change. Lint ok.

Review round (27c2eca): _variables included, rebuild gated on the option, standard integration test plus unit tests over a stubbed host (test/unit/event-command-variable-mutations-only.test.js).

Suite Result
test-integration, installed sandbox, option off 1919 passing, 0 failing
test-integration, #1116 branch, option forced on 1919 passing, 0 failing
test-integration-legacy, #1116 branch 33 passing
test-unit 576 passing. 1 failing: version.test.js, local node_modules vs lockfile, not this change
lint ok

Additional notes

Draft to develop. Hold it until the sandbox release, then add the dependency bump.

Generated with Claude Code

Pass variableMutationsOnly to host.execute so the sandbox sends back only
mutations for environment, globals and collectionVariables. The full
post-script scopes for the script trigger are rebuilt from the context
sent to the sandbox (the masked clone when forbidden secrets apply) with
the mutations applied. An older sandbox ignores the option and sends the
full scopes, which are used as before.

Co-Authored-By: Claude <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.65%. Comparing base (4e40645) to head (388d0a6).

❌ Your project check has failed because the head coverage (34.08%) is below the target coverage (35.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1568      +/-   ##
===========================================
+ Coverage    77.61%   77.65%   +0.03%     
===========================================
  Files           51       51              
  Lines         4361     4368       +7     
  Branches      1234     1234              
===========================================
+ Hits          3385     3392       +7     
  Misses         732      732              
  Partials       244      244              
Flag Coverage Δ
integration 68.86% <86.66%> (+<0.01%) ⬆️
legacy 34.08% <46.66%> (-0.08%) ⬇️
unit 47.00% <100.00%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Kart03

Kart03 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Full-run results, for reviewers. This was a copy of the PPC Governor collection (AUTO-1062) through a local CLI build: 44 VUs, fixed load, 20 min, 6 GiB / 4 CPU. The run used a prototype that patched the sandbox the same way as this PR (only mutations back for environment, globals and collectionVariables), and the comparison run was identical apart from that.

Full scopes back Mutations only
Per script run, ~4 MB of variables (micro-benchmark) 22.5 ms, 22.4 MB allocated 13.3 ms, 14.0 MB allocated
Worker peak 2.65 GB 1.92 GB
Worker avg, minutes 4–12 / after the ACS step 1.20 / 2.23 GB 0.97 / 1.58 GB
Container peak 3.45 GB 2.81 GB
rps after the ACS step 31.7 40.0
Requests outside the time-bound loop 80,828 80,828
Assertions failed / passed 220 / 21,648 220 / 21,648

Why it helps: the allocation profile of the host thread shows the uvm bridge making about 6 copies of the variables per script run (Flatted.stringify + join + postMessage + receive + Flatted.parse), with the sandbox thread mirroring them. One sandbox thread serves all VUs, so the copies also capped throughput. This removes the return trip.

Reviewer guide covering all the AUTO-1062 changes: https://claude.ai/artifact/S7vBMoFxTA9kjzQHhQTXpt

@Kart03
Kart03 marked this pull request as ready for review October 15, 2026 14:54
Kart03 and others added 3 commits October 21, 2026 19:03
Co-Authored-By: Claude <noreply@anthropic.com>
Read script.variableMutationsOnly from run options instead of always
asking the sandbox for mutations only, so existing consumers keep the
full-scope round trip unless they opt in.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread lib/runner/extensions/event.command.js Outdated
'request', 'response'],

// variable scopes the sandbox sends back as mutations alone, see `variableMutationsOnly`
MUTATIONS_ONLY_SCOPES = ['environment', 'globals', 'collectionVariables'],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is _variables (local variable scope) not included?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocker, I was keeping the change narrow. It's included now in 27c2eca: _variables is rebuilt from mutations like the other three scopes, before payload.context, state and the partition read it, and the sandbox sends it as mutations too (#1116 now uses CONTEXT_VARIABLE_SCOPES).

Comment thread lib/runner/extensions/event.command.js Outdated
Comment thread lib/runner/extensions/event.command.js Outdated
Comment thread CHANGELOG.yaml Outdated
Comment thread README.md Outdated
Comment thread test/integration/sanity/variable-mutations-only.test.js Outdated
Kart03 and others added 3 commits October 22, 2026 13:02
- include _variables in the scopes rebuilt from mutations
- rebuild only when the run asked for mutations only
- use const in applyScopeMutations
- list the changelog entry under new features, trim the docs note
- replace the sandbox-importing integration test with a standard one
  plus unit tests over a stubbed host

Co-Authored-By: Claude <noreply@anthropic.com>
postman-sandbox v6.9.0 adds the `variableMutationsOnly` execute option
(postmanlabs/postman-sandbox#1116) that `script.variableMutationsOnly`
relies on.

Co-Authored-By: Claude <noreply@anthropic.com>
Cover the two branches of applyScopeMutations that no test reached: a
scope the run holds as a plain object (item command falls back to `{}`
for a scope missing from its payload) and a scope the sandbox sends back
without mutations.

Co-Authored-By: Claude <noreply@anthropic.com>
@Kart03
Kart03 merged commit 60482b6 into develop Oct 22, 2026
25 of 27 checks passed
@Kart03
Kart03 deleted the feat/mutations-only-result branch October 22, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants