Skip to content

fix: keep statements that share a line in one PythonCodeSplitter chunk - #13752

Open
1AbdullahButt wants to merge 2 commits into
deepset-ai:mainfrom
1AbdullahButt:fix/python-code-splitter-same-line-statements
Open

1AbdullahButt wants to merge 2 commits into
deepset-ai:mainfrom
1AbdullahButt:fix/python-code-splitter-same-line-statements

Conversation

@1AbdullahButt

Copy link
Copy Markdown

Related Issues

Proposed Changes:

_extract_units slices every top-level node from cursor to its end_lineno. When a statement shares its line with the previous one (x = 1; y = 2), the previous unit already moved cursor past that line, so the second statement became a unit with an empty source. That unit turned into an empty chunk at the end of a file, or was merged into the next chunk and set its start_line to the previous line.

The loop now skips a node whose last line is before cursor, since its text is already in the previous unit. Only simple statements can follow a ;, so this covers the catch-all statement branch and import blocks (x = 1; import os).

How did you test it?

  • Added test_statements_sharing_a_line_stay_in_one_chunk (two cases: the shared line at the end of the file, and before a function). Both fail on main and pass with the fix.
  • pytest test/components/preprocessors/test_python_code_splitter.py: 87 passed.
  • ruff check and ruff format --check on the changed files pass.
  • I did not run the full hatch run test:unit suite or mypy (hatch isn't installed on this machine; I used a plain venv).

Notes for the reviewer

Chunk content is unchanged for these inputs; only the empty chunk disappears and start_line of the following chunk is correct again.

This PR was fully generated with an AI assistant (Claude Code). I have reviewed the changes and run the relevant tests.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

🤖 Generated with Claude Code

A top-level statement on the same line as the previous one (x = 1; y = 2)
was turned into a unit with an empty source. It became an empty chunk at
the end of the file, or was merged into the next chunk and moved that
chunk's start_line back to the previous line. Skip such nodes, since their
text is already part of the previous unit.

Fixes deepset-ai#13751

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@1AbdullahButt
1AbdullahButt requested a review from a team as a code owner November 14, 2026 18:03
@1AbdullahButt
1AbdullahButt requested review from davidsbatista and removed request for a team November 14, 2026 18:03
@vercel

vercel Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@1AbdullahButt is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @1AbdullahButt, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from davidsbatista November 14, 2026 19:21
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 14, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 14, 2026 19:21
@HaystackBot
HaystackBot marked this pull request as ready for review November 14, 2026 20:52
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Nov 14, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @1AbdullahButt! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

Comment on lines +353 to +357
# A statement that shares its line with the previous one (e.g. `x = 1; y = 2`) is already part of the
# previous unit's source, so it must not become an empty unit of its own.
if (node.end_lineno or node.lineno) < cursor:
node_idx += 1
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this still breaks when the statement after the ; runs onto more lines, e.g. x = 1; y = dict(\n a=1,\n) with max_effective_lines=1 gives a chunk that ends at y = dict( and a second chunk a=1,\n) with start_line 1. letting the previous unit take the rest of those lines fixes it:

Suggested change
# A statement that shares its line with the previous one (e.g. `x = 1; y = 2`) is already part of the
# previous unit's source, so it must not become an empty unit of its own.
if (node.end_lineno or node.lineno) < cursor:
node_idx += 1
continue
# A statement that starts on the line the previous unit ended on (e.g. `x = 1; y = 2`) is already part
# of that unit's source. If it runs onto later lines, the previous unit takes those lines too.
if node.lineno < cursor:
end = node.end_lineno or node.lineno
if end >= cursor and units:
units[-1].source += self._slice_lines(source_lines, cursor, end)
units[-1].end_line = end
cursor = end + 1
node_idx += 1
continue

fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@9911d85

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. I cherry-picked your commit as-is (59c6b38, authorship kept), so a statement after the ; that runs onto later lines now stays in the previous chunk. All 89 splitter tests pass locally, including your two new cases.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PythonCodeSplitter emits an empty chunk or a wrong start_line for statements that share a line (x = 1; y = 2)

4 participants