Skip to content

fix(parquet/file): count rows from repetition levels for V1 data pages - #1640

Open
EdwardShen125 wants to merge 1 commit into
apache:mainfrom
EdwardShen125:fix/1522-v1-seek-row-count
Open

EdwardShen125 wants to merge 1 commit into
apache:mainfrom
EdwardShen125:fix/1522-v1-seek-row-count

Conversation

@EdwardShen125

Copy link
Copy Markdown
Contributor

Rationale for this change

SeekToRow on a repeated column returns wrong rows without an error when
the file has V1 data pages and no page index (#1522). Thanks to @hcrosse
for the root cause and repro: serializedPageReader.Next advances
rowsSeen by a V1 page's num_values, which counts leaf values, not
records. On his 300-row list<int32> repro, SeekToRow(150) returned 243
rows starting at [195, 196, 197].

What changes are included in this PR?

A V1 page's records are now counted from its level-0 repetition levels
when the column repeats; flat columns keep the num_values fast path.
rowsSeen drives both the seek loop and the chunk-termination check.

Records can span page boundaries, so the page a seek lands on can start
with the tail of a record a previous page already counted. skipRows
discards that tail now. The repetition level buffer is reset per page (V2
already did this), so a seek can't carry one page's levels into the next.

Encrypted page readers set maxRepLevel/maxDefLevel like the
unencrypted one, so they get the same count.

The cost: repetition levels on V1 repeated pages are decoded twice, once
for the count and once for the values.

Are these changes tested?

TestRecordReaderSeekToRowRepeatedV1WithoutPageIndex writes list<int32>
with tiny pages and no page index, seeks to rows 0/1/150/299 for both page
versions, and reads to the end. The V1 legs fail without the change and
pass with it.

go test -count=1 ./parquet/file/... ./parquet/pqarrow/...
go test -race -count=1 -run 'SeekToRow' ./parquet/file/... ./parquet/pqarrow/...

Are there any user-facing changes?

SeekToRow returns the requested rows for repeated V1 columns without a
page index, instead of silently wrong ones. No API change.

SeekToRow on a repeated column returned wrong rows without an error
when the file used V1 data pages and no page index. A V1 page header
only carries num_values, which counts leaf values (levels) rather
than records, so rowsSeen advanced by the leaf count: seeks landed
early and the chunk-termination check cut reads short.

Count the records of a V1 page from its level-0 repetition levels
when the column repeats, keeping the num_values fast path for flat
columns. The page reader constructors for encrypted columns now set
the level descriptors too, so they get the same count.

Records may also span a page boundary, so a page can begin with the
tail of a record a previous page already counted; skipRows now
discards that leading partial, and the repetition level buffer is
reset and registered per page so a seek cannot leak the previous
page's levels into the next one.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Counting V1 rows from the repetition levels is a good fix, and it means a seek lands on the right page. I ran a randomized test with an optional list<optional int32> column across V1 and V2 pages, page index on and off, and dictionary on and off, 300 random sequences per setup. Skips and seeks followed by reads returned the right rows in every sequence. The rest of ./parquet/file passes with -race -tags assert,test.

Two changes are needed, inline. The first makes CI fail.

This overlaps #1608, which fixes the same two problems (skips across a row that spans pages, and seeks without an offset index) with a different approach, and also fixes a third: ReadBatch with nil repetition levels leaves the repetition decoder behind, so a following read or skip on that page gets the wrong levels. This PR doesn't cover that case, which still fails on both V1 and V2. The two PRs conflict in column_reader.go and the tests. I'd prefer to land this one first and then rebase #1608 down to the nil repetition level fix and its tests.

Once #1522 is fixed, the readThroughGap workaround that #1632 added in pqarrow can be removed.

}
totalRows++
}
rec.Release()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read() returns the reader's current batch, and the reader releases it on the next call, so this is a second release. CI runs with -tags assert,test (ci/scripts/test.sh), and with those tags this panics with too many releases in page-v1/seek=0. Without the assert tag it passes. Please remove this Release.

func (p *serializedPageReader) countRowsV1(levels []byte, repEnc format.Encoding, numValues int32) (int64, error) {
var (
dec encoding.LevelDecoder
batch [4096]int16

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs for every V1 page of a repeated column, on ordinary sequential reads as well as seeks. Because batch[:...] is passed through Decode, the 8 KiB array is most likely heap-allocated on each call. Reading a 512K-row list<int32> V1 column showed no measurable time change, but allocations went up 5.3% (+18 per read). A scratch buffer kept on serializedPageReader would avoid that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants