Skip to content

fix: preserve pipeline input dictionaries across runs and streams - #13619

Merged
sjrl merged 3 commits into
deepset-ai:mainfrom
remote-controlled-man:codex/prflow-6dd8b1edbb5a
Nov 10, 2026
Merged

sjrl merged 3 commits into
deepset-ai:mainfrom
remote-controlled-man:codex/prflow-6dd8b1edbb5a

Conversation

@remote-controlled-man

@remote-controlled-man remote-controlled-man commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

No existing issue found during targeted searches.

Proposed Changes

Build a new component-input mapping in the shared preparation used by Pipeline.run(), Pipeline.run_async(), and Pipeline.stream(), preserving the caller's data dictionary and the existing deep-copy behavior. Concurrent streams that reuse the same input dictionary keep their chunks isolated. Remove the stream-only outer copy.

Describe these user-visible changes in the release note.

How did you test it?

The public-API regressions exercise run() and run_async() with component-qualified inputs and a component that mutates its own input. Both cases fail with the original mutating helper and pass with the shared fix. The original concurrent-stream regression still covers runs with and without a runtime callback.

All four focused cases pass without model calls, mocks, or timing sleeps. On commit 7573220, using the project Hatch environments:

  • hatch run test:unit test/core/pipeline: 455 passed, 155 integration tests deselected.
  • hatch -e test run mypy --cache-dir=.mypy_cache/ on the four relevant Python files: passed.
  • hatch run fmt-check on the four relevant Python files: passed.
  • hatch run pre-commit run --files on the five relevant files: passed, with inapplicable hooks skipped.

Reno lint is validated by the Check Release Notes CI workflow, which uses a full checkout. Local execution remains unavailable in this shallow checkout; release-note YAML and reStructuredText checks run through pre-commit.

The full repository suite and external-service integration tests were not run locally.

Notes for the reviewer

This PR was fully generated with OpenAI Codex while investigating pipeline execution isolation. A separate agent reviewed the frozen changes and validation evidence.

Checklist

  • Read the contributing guidelines and code of conduct.
  • Added regression tests and a release note.
  • Used a conventional commit title.
  • Ran the pre-commit hooks.
  • No existing related issue to update.
  • Existing public signatures are unchanged.

@remote-controlled-man
remote-controlled-man requested a review from a team as a code owner November 8, 2026 09:53
@remote-controlled-man
remote-controlled-man requested review from sjrl and a balanced review from Copilot and removed request for a team November 8, 2026 09:53
@vercel

vercel Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

@remote-controlled-man is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 8, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @remote-controlled-man, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from sjrl November 8, 2026 11:21
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 8, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 8, 2026 11:21
@HaystackBot
HaystackBot marked this pull request as ready for review November 8, 2026 13:10
@HaystackBot
HaystackBot requested a review from sjrl November 8, 2026 13:10
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Nov 8, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @remote-controlled-man! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@sjrl sjrl self-assigned this Nov 9, 2026
@github-actions github-actions Bot added the type:documentation Improvements on the docs label Nov 10, 2026
Comment thread haystack/core/pipeline/pipeline.py Resolved
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/core/pipeline
  base.py
  pipeline.py
Project Total  

This report was generated by python-coverage-comment-action

@remote-controlled-man remote-controlled-man changed the title fix: isolate concurrent pipeline streams sharing input data fix: preserve pipeline input dictionaries across runs and streams Nov 10, 2026


@pytest.mark.parametrize("use_async", [False, True], ids=["run", "run_async"])
@pytest.mark.parametrize("flat_input", [False, True], ids=["nested", "flat"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets drop the flat_input parametrization. The flat path already builds a new dict, so those cases pass on main too.

Comment on lines 1281 to 1282
# deepcopying the inputs prevents the Pipeline run logic from being altered unexpectedly
# when the same input reference is passed to multiple components.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we extend this comment to say we build a new dict so the caller's data is left unmodified?

@sjrl sjrl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sjrl
sjrl merged commit 7c59972 into deepset-ai:main Nov 10, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants