Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… state Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlushCurrentPage with no buffered values wrote a zero-value data page. Return early instead so callers can flush between content-defined chunks without producing empty pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rtions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xed-width types Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oll, wrap option errors Replaces the runtime crypto/md5 computation, which panics in FIPS 140-only mode, with a generated table. Keeps the rolling hash in locals in the per-byte loop, fixes single-subtest leaks in tests, and wraps option errors in arrow.ErrInvalid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…name error test, document CDC options Adds race-detector build-tagged constants, a reduced type matrix for TestCDCSingleRowGroup under -race, documents NormLevel and the Go-specific CDC behaviors, and drops the STRING condition on the plainPages assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e chunking loop Keep the rolling state in a by-value struct local to Chunks, merge the three level loops into one, and roll 1/2/4/8-byte values as unrolled words inline. Chunk boundaries are unchanged (pinned by TestChunksStableBoundaries). BenchmarkChunks, 1Mi levels, DefaultCDCOptions, Apple M4 (median ms): required/int64 5.7 -> 3.5 nullable/int64 7.8 -> 4.1 nested/list<int32> 10.3 -> 5.2 required/string 7.5 -> 8.3 The per-value byte path (strings/binary) got about 0.8 ns/value slower. End to end, pqarrow writes of 1Mi rows with dictionary on: the 7-column primitive CDC write went from 39.6 to 31.3 ms, the string CDC write from 33.5 to 34.7 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- writeArrowToColumn: drop the unreachable len(chunks)==0 branch and resolve FlushCurrentPage once before the chunk loop. - Have the gear table generator emit numGearTables and remove gearhash.go; the table values are unchanged. - Document why newValueSource matches binary-like arrays by concrete type (an interface case is about 10% slower on strings). - Tests: table-driven edit scenarios and mask cases, shared input and reader helpers, array.NewTableFromSlice for table assembly, and the multi row group tables generated once. Test names and assertions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Chunks returns arrow.ErrInvalid when the levels can't cover the values, instead of returning no chunks and silently dropping the data. - Hash fixed-width values as their little-endian bytes so chunk boundaries don't depend on the host's byte order. - Skip the dictionary comparison for slices sharing the preserved dictionary's data, so CDC chunks don't compare it once per chunk. - Release each chunk's value slice in a deferred call. - WriteArrowToColumn returns arrow.ErrNotImplemented when CDC is enabled. - Add FlushCurrentPage to file.ColumnChunkWriter. - Compile the chunk validation only with the assert build tag. - Wrap the unsupported-type error with arrow.ErrNotImplemented. - Document that chunking state spans row groups, as in Arrow C++ 27. - Tests: test through the public API, pin the gear tables to the C++ header, and cover decimals in the word/byte equivalence test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share the round-trip and page-size helpers across the CDC tests, use existing arrow helpers (NewTableReader, ListLikeType, TotalArraySize, mockpagewriter) instead of hand-written equivalents, and inline helpers used once. Validate chunks against the real value count in assert builds and convert decimal words to little-endian on every host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hey @kszucs! This has been on my radar and list of stuff to do for a while, ever since I saw your blog post about it. So I'm super excited to take a look at this. Thanks much! |
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, this is a careful port. I compared Chunks against the C++ chunker. The min-size skip, the 8-match/max-size cut, cutting before the triggering value, hashing levels as 2 little-endian bytes, never resetting the hash, and cutting nested columns only at rep == 0 all match. Null slots hash only their definition level, and value slots line up with what pqarrow slices per chunk. The FlushCurrentPage early return is safe because numBufferedValues counts levels, so all-null pages are still written. The dictionary shortcut only skips the comparison when both sides share the same *Data.
Locally (Apple M4, Go 1.27.1), go test -tags assert passes for parquet/internal/cdc, parquet/pqarrow and parquet/file, -race passes for cdc and pqarrow, and go vet passes for s390x.
API break. Adding FlushCurrentPage() error to the exported file.ColumnChunkWriter interface breaks implementations outside this module in a v18 minor release. Since every column writer here already has the method, writeArrowToColumn could use an optional interface instead, e.g. cw.(interface{ FlushCurrentPage() error }), and return an error if it's missing. That keeps the interface unchanged. The other two listed changes (no empty flushes, WriteArrowToColumn rejecting CDC) seem fine.
Non-blocking:
- Fixed-width values of 8 bytes take the native-word path. For multi-field types such as
DAY_TIMEintervals that gives a different byte order on big-endian hosts than on little-endian ones, so the byte-order-independence claim doesn't hold there.pqarrowdoesn't write those types today, so a comment, or limiting the word path to single-field types, would do. - The boolean path builds
[]byte{1}/[]byte{0}per value; two package-level arrays would avoid it. - A small fixture written by PyArrow with CDC would pin cross-implementation compatibility, which the description notes isn't covered yet.
- At ~5.3k lines (most of it generated tables and tests) this is reviewable, but splitting the
filepackage change into its own commit/PR would make history easier to follow.
I'm also looking at the write overhead with CDC enabled (about 3x for int64 and 2x for strings in BenchmarkWriteContentDefinedChunking here) and will follow up with any profiling results.
|
@kszucs following up on the write overhead with CDC enabled. I profiled it and prototyped some changes to Where the time goes (pprof, Apple M4): in Changes that helped:
SIMD: I don't think it's worth it here. The gear hash has to stay as is for compatibility with C++ and arrow-rs, and vectorizing it needs 64-bit gathers. The FAST'25 VectorCDC paper (Udayashankar et al.) measured only ~1.2x for AVX-512 Gear and no gain for FastCDC, because splitting hashing from boundary detection gives up min-size skipping. NEON has no 64-bit gather, so arm64 wouldn't benefit at all. Other ideas, not prototyped:
None of this blocks the PR. Happy to share the patch (~200 lines, |
|
Thanks @zeroshade for reviewing and optimizing! I'm busy with the java/cpp/rust implementations but circling back as soon as I can. |
Rationale for this change
Content-defined chunking (CDC) makes the Parquet writer cut data pages at boundaries derived from the data instead of from a byte count. When a table is edited (rows inserted, deleted or updated) and written again, most data pages come out byte-identical to the previous version, so content-addressable storage such as Hugging Face Xet only has to store the pages that changed.
Arrow C++/PyArrow (apache/arrow#45360) and arrow-rs (apache/arrow-rs#9450) already support it. This PR ports the same algorithm to arrow-go.
What changes are included in this PR?
Writer option:
parquet.CDCOptions,parquet.DefaultCDCOptions()(256 KiB min, 1 MiB max, norm level 0),parquet.WithContentDefinedChunkingandWriterProperties.ContentDefinedChunking(). CDC is off by default.Chunker: a new internal package,
parquet/internal/cdc, ports the C++ContentDefinedChunker. It keeps:Nested columns are cut only at record boundaries. Fixed-width values are hashed as their little-endian bytes, so boundaries don't depend on the host's byte order. The gear tables are checked in as
gearhash_table.go, generated bygearhash_gen.gothroughgo generate.pqarrow integration:
pqarrow.FileWriterkeeps one chunker per leaf column for the life of the file. Page boundaries therefore don't depend on how rows are split across row groups or acrossWrite,WriteBuffered,WriteTableandWriteColumnChunked/WriteColumnDatacalls.NewFileWriterreturn an error before anything is written to the sink.filepackage:ColumnChunkWritergainsFlushCurrentPage() error, which every column writer already implements.FlushCurrentPageno longer writes an empty data page when nothing is buffered.Dictionary writes: slices of one dictionary array share their dictionary data, so
pqarrownow skips the dictionary comparison for them.Behaviour that differs from Arrow C++:
pqarrow.FileWriterchunks.pqarrow.WriteArrowToColumnreturnsarrow.ErrNotImplementedwhen CDC is enabled, and the low-levelfilecolumn writers ignore the option. C++ throws fromWriteBatch.Page boundaries are not compared against files written by PyArrow or arrow-rs. The algorithm and gear tables are the same, but there are no cross-implementation fixtures.
Of the ~5.3k added lines, about 2.1k are the generated gear table and about 2.5k are tests.
Are these changes tested?
Yes.
parquet/internal/cdc:TestChunksStableBoundariespins exact boundaries, so a hashing change can't silently break deduplication between versions.-tags assert.parquet/pqarrow: a port of the C++ CDC test suite, including its data generator and thePagesFollowContentpage assertion.WriteBuffered, column-by-column writes or dictionary-typed input.WriteArrowToColumn.-racethe type matrix shrinks to the subset C++ runs under sanitizers.parquet/file: aFlushCurrentPagecall with nothing buffered writes no page.BenchmarkWriteContentDefinedChunkingcovers int64 and string columns with CDC off and on.BenchmarkChunkscovers the chunker on its own.Local runs:
go test -tags assertand-tags assert,noasmover./parquet/...;-raceon thecdc,pqarrowandfilepackages;-tags pqarrow_read_onlybuild;go vetof the changed packages for 386 and s390x;parquet.Are there any user-facing changes?
Yes. CDC is a new, experimental writer option:
This PR includes breaking changes to public APIs:
file.ColumnChunkWriterhas a new method,FlushCurrentPage() error, so implementations outside this module must add it.FlushCurrentPagewith nothing buffered no longer writes an empty data page.pqarrow.WriteArrowToColumnreturns an error when the column writer's properties enable CDC.🤖 Generated with Claude Code