Repository navigation
fix: keep statements that share a line in one PythonCodeSplitter chunk - #13752
1AbdullahButt wants to merge 2 commits into
Conversation
A top-level statement on the same line as the previous one (x = 1; y = 2) was turned into a unit with an empty source. It became an empty chunk at the end of the file, or was merged into the next chunk and moved that chunk's start_line back to the previous line. Skip such nodes, since their text is already part of the previous unit. Fixes deepset-ai#13751 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@1AbdullahButt is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @1AbdullahButt, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
|
Thanks for signing the CLA, @1AbdullahButt! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
| # A statement that shares its line with the previous one (e.g. `x = 1; y = 2`) is already part of the | ||
| # previous unit's source, so it must not become an empty unit of its own. | ||
| if (node.end_lineno or node.lineno) < cursor: | ||
| node_idx += 1 | ||
| continue |
There was a problem hiding this comment.
this still breaks when the statement after the ; runs onto more lines, e.g. x = 1; y = dict(\n a=1,\n) with max_effective_lines=1 gives a chunk that ends at y = dict( and a second chunk a=1,\n) with start_line 1. letting the previous unit take the rest of those lines fixes it:
| # A statement that shares its line with the previous one (e.g. `x = 1; y = 2`) is already part of the | |
| # previous unit's source, so it must not become an empty unit of its own. | |
| if (node.end_lineno or node.lineno) < cursor: | |
| node_idx += 1 | |
| continue | |
| # A statement that starts on the line the previous unit ended on (e.g. `x = 1; y = 2`) is already part | |
| # of that unit's source. If it runs onto later lines, the previous unit takes those lines too. | |
| if node.lineno < cursor: | |
| end = node.end_lineno or node.lineno | |
| if end >= cursor and units: | |
| units[-1].source += self._slice_lines(source_lines, cursor, end) | |
| units[-1].end_line = end | |
| cursor = end + 1 | |
| node_idx += 1 | |
| continue |
fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@9911d85
There was a problem hiding this comment.
Thanks, good catch. I cherry-picked your commit as-is (59c6b38, authorship kept), so a statement after the ; that runs onto later lines now stays in the previous chunk. All 89 splitter tests pass locally, including your two new cases.
Related Issues
Proposed Changes:
_extract_unitsslices every top-level node fromcursorto itsend_lineno. When a statement shares its line with the previous one (x = 1; y = 2), the previous unit already movedcursorpast that line, so the second statement became a unit with an empty source. That unit turned into an empty chunk at the end of a file, or was merged into the next chunk and set itsstart_lineto the previous line.The loop now skips a node whose last line is before
cursor, since its text is already in the previous unit. Only simple statements can follow a;, so this covers the catch-all statement branch and import blocks (x = 1; import os).How did you test it?
test_statements_sharing_a_line_stay_in_one_chunk(two cases: the shared line at the end of the file, and before a function). Both fail onmainand pass with the fix.pytest test/components/preprocessors/test_python_code_splitter.py: 87 passed.ruff checkandruff format --checkon the changed files pass.hatch run test:unitsuite or mypy (hatch isn't installed on this machine; I used a plain venv).Notes for the reviewer
Chunk content is unchanged for these inputs; only the empty chunk disappears and
start_lineof the following chunk is correct again.This PR was fully generated with an AI assistant (Claude Code). I have reviewed the changes and run the relevant tests.
Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.🤖 Generated with Claude Code