Repository navigation
fix(parquet/file): count rows from repetition levels for V1 data pages - #1640
EdwardShen125 wants to merge 1 commit into
Conversation
SeekToRow on a repeated column returned wrong rows without an error when the file used V1 data pages and no page index. A V1 page header only carries num_values, which counts leaf values (levels) rather than records, so rowsSeen advanced by the leaf count: seeks landed early and the chunk-termination check cut reads short. Count the records of a V1 page from its level-0 repetition levels when the column repeats, keeping the num_values fast path for flat columns. The page reader constructors for encrypted columns now set the level descriptors too, so they get the same count. Records may also span a page boundary, so a page can begin with the tail of a record a previous page already counted; skipRows now discards that leading partial, and the repetition level buffer is reset and registered per page so a seek cannot leak the previous page's levels into the next one.
zeroshade
left a comment
There was a problem hiding this comment.
Thanks. Counting V1 rows from the repetition levels is a good fix, and it means a seek lands on the right page. I ran a randomized test with an optional list<optional int32> column across V1 and V2 pages, page index on and off, and dictionary on and off, 300 random sequences per setup. Skips and seeks followed by reads returned the right rows in every sequence. The rest of ./parquet/file passes with -race -tags assert,test.
Two changes are needed, inline. The first makes CI fail.
This overlaps #1608, which fixes the same two problems (skips across a row that spans pages, and seeks without an offset index) with a different approach, and also fixes a third: ReadBatch with nil repetition levels leaves the repetition decoder behind, so a following read or skip on that page gets the wrong levels. This PR doesn't cover that case, which still fails on both V1 and V2. The two PRs conflict in column_reader.go and the tests. I'd prefer to land this one first and then rebase #1608 down to the nil repetition level fix and its tests.
Once #1522 is fixed, the readThroughGap workaround that #1632 added in pqarrow can be removed.
| } | ||
| totalRows++ | ||
| } | ||
| rec.Release() |
There was a problem hiding this comment.
Read() returns the reader's current batch, and the reader releases it on the next call, so this is a second release. CI runs with -tags assert,test (ci/scripts/test.sh), and with those tags this panics with too many releases in page-v1/seek=0. Without the assert tag it passes. Please remove this Release.
| func (p *serializedPageReader) countRowsV1(levels []byte, repEnc format.Encoding, numValues int32) (int64, error) { | ||
| var ( | ||
| dec encoding.LevelDecoder | ||
| batch [4096]int16 |
There was a problem hiding this comment.
This runs for every V1 page of a repeated column, on ordinary sequential reads as well as seeks. Because batch[:...] is passed through Decode, the 8 KiB array is most likely heap-allocated on each call. Reading a 512K-row list<int32> V1 column showed no measurable time change, but allocations went up 5.3% (+18 per read). A scratch buffer kept on serializedPageReader would avoid that.
Rationale for this change
SeekToRowon a repeated column returns wrong rows without an error whenthe file has V1 data pages and no page index (#1522). Thanks to @hcrosse
for the root cause and repro:
serializedPageReader.NextadvancesrowsSeenby a V1 page'snum_values, which counts leaf values, notrecords. On his 300-row
list<int32>repro,SeekToRow(150)returned 243rows starting at
[195, 196, 197].What changes are included in this PR?
A V1 page's records are now counted from its level-0 repetition levels
when the column repeats; flat columns keep the
num_valuesfast path.rowsSeendrives both the seek loop and the chunk-termination check.Records can span page boundaries, so the page a seek lands on can start
with the tail of a record a previous page already counted.
skipRowsdiscards that tail now. The repetition level buffer is reset per page (V2
already did this), so a seek can't carry one page's levels into the next.
Encrypted page readers set
maxRepLevel/maxDefLevellike theunencrypted one, so they get the same count.
The cost: repetition levels on V1 repeated pages are decoded twice, once
for the count and once for the values.
Are these changes tested?
TestRecordReaderSeekToRowRepeatedV1WithoutPageIndexwriteslist<int32>with tiny pages and no page index, seeks to rows 0/1/150/299 for both page
versions, and reads to the end. The V1 legs fail without the change and
pass with it.
Are there any user-facing changes?
SeekToRowreturns the requested rows for repeated V1 columns without apage index, instead of silently wrong ones. No API change.