Skip to content

feat(parquet/pqarrow): add content-defined chunking - #1469

Draft
kszucs wants to merge 17 commits into
apache:mainfrom
kszucs:parquet-cdc
Draft

kszucs wants to merge 17 commits into
apache:mainfrom
kszucs:parquet-cdc

Conversation

@kszucs

@kszucs kszucs commented Nov 5, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Content-defined chunking (CDC) makes the Parquet writer cut data pages at boundaries derived from the data instead of from a byte count. When a table is edited (rows inserted, deleted or updated) and written again, most data pages come out byte-identical to the previous version, so content-addressable storage such as Hugging Face Xet only has to store the pages that changed.

Arrow C++/PyArrow (apache/arrow#45360) and arrow-rs (apache/arrow-rs#9450) already support it. This PR ports the same algorithm to arrow-go.

What changes are included in this PR?

  • Writer option: parquet.CDCOptions, parquet.DefaultCDCOptions() (256 KiB min, 1 MiB max, norm level 0), parquet.WithContentDefinedChunking and WriterProperties.ContentDefinedChunking(). CDC is off by default.

  • Chunker: a new internal package, parquet/internal/cdc, ports the C++ ContentDefinedChunker. It keeps:

    • the gearhash rolling hash over (def level, rep level, value);
    • the same 8 gear tables and mask formula;
    • a cut after 8 mask matches, one per gear table, or at the max size.

    Nested columns are cut only at record boundaries. Fixed-width values are hashed as their little-endian bytes, so boundaries don't depend on the host's byte order. The gear tables are checked in as gearhash_table.go, generated by gearhash_gen.go through go generate.

  • pqarrow integration: pqarrow.FileWriter keeps one chunker per leaf column for the life of the file. Page boundaries therefore don't depend on how rows are split across row groups or across Write, WriteBuffered, WriteTable and WriteColumnChunked/WriteColumnData calls.

    • Each leaf write is sliced per chunk, zero-copy, and the current page is closed between chunks.
    • Invalid options make NewFileWriter return an error before anything is written to the sink.
  • file package: ColumnChunkWriter gains FlushCurrentPage() error, which every column writer already implements. FlushCurrentPage no longer writes an empty data page when nothing is buffered.

  • Dictionary writes: slices of one dictionary array share their dictionary data, so pqarrow now skips the dictionary comparison for them.

Behaviour that differs from Arrow C++:

  • State across row groups: chunker state carries over across row groups. C++ does the same since GH-51684 (Arrow 27.0.0); earlier versions restart at every row group.
  • Nested columns: a boundary at the very start of a write call closes the page left open by the previous call. C++ drops that boundary, which makes nested pages depend on how the data was split across write calls.
  • Dictionary fallback: falling back from dictionary to plain encoding doesn't start a new page, because Go's fallback re-encodes the buffered page.
  • Writer scope: only pqarrow.FileWriter chunks. pqarrow.WriteArrowToColumn returns arrow.ErrNotImplemented when CDC is enabled, and the low-level file column writers ignore the option. C++ throws from WriteBatch.

Page boundaries are not compared against files written by PyArrow or arrow-rs. The algorithm and gear tables are the same, but there are no cross-implementation fixtures.

Of the ~5.3k added lines, about 2.1k are the generated gear table and about 2.5k are tests.

Are these changes tested?

Yes.

  • parquet/internal/cdc:
    • The mask and option-validation tables and the gear-table values are taken from the C++ tests.
    • Chunk invariants are checked for required, nullable and nested columns, along with the forced cut at the max size and determinism.
    • Chunking a column in one call gives the same boundaries as chunking it across two calls, including when a boundary falls exactly at the split.
    • Type dispatch: dictionaries chunk like their indices, extensions like their storage, all binary-like types alike, and unsupported types return an error.
    • Hashing a fixed-width value word by word matches hashing its little-endian bytes.
    • TestChunksStableBoundaries pins exact boundaries, so a hashing change can't silently break deduplication between versions.
    • Invariant checks run in builds with -tags assert.
  • parquet/pqarrow: a port of the C++ CDC test suite, including its data generator and the PagesFollowContent page assertion.
    • Single row group: inserts, deletes, updates, prepend, append, an empty table and array offsets. The type matrix covers dictionary on and off and data page V1/V2.
    • Multiple row groups: edits across row groups.
    • Independence: pages don't depend on row-group boundaries, write calls, Arrow chunking, WriteBuffered, column-by-column writes or dictionary-typed input.
    • Error paths: invalid options, write errors, missing levels and WriteArrowToColumn.
    • Under -race the type matrix shrinks to the subset C++ runs under sanitizers.
  • parquet/file: a FlushCurrentPage call with nothing buffered writes no page.
  • Benchmarks: BenchmarkWriteContentDefinedChunking covers int64 and string columns with CDC off and on. BenchmarkChunks covers the chunker on its own.

Local runs:

  • go test -tags assert and -tags assert,noasm over ./parquet/...;
  • -race on the cdc, pqarrow and file packages;
  • a -tags pqarrow_read_only build;
  • go vet of the changed packages for 386 and s390x;
  • golangci-lint over parquet.

Are there any user-facing changes?

Yes. CDC is a new, experimental writer option:

props := parquet.NewWriterProperties(
	parquet.WithContentDefinedChunking(parquet.DefaultCDCOptions()),
)
fw, err := pqarrow.NewFileWriter(schema, w, props, pqarrow.DefaultWriterProps())

This PR includes breaking changes to public APIs:

  • file.ColumnChunkWriter has a new method, FlushCurrentPage() error, so implementations outside this module must add it.
  • Calling FlushCurrentPage with nothing buffered no longer writes an empty data page.
  • pqarrow.WriteArrowToColumn returns an error when the column writer's properties enable CDC.

🤖 Generated with Claude Code

kszucs and others added 17 commits November 5, 2026 14:22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlushCurrentPage with no buffered values wrote a zero-value data page.
Return early instead so callers can flush between content-defined
chunks without producing empty pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rtions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xed-width types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oll, wrap option errors

Replaces the runtime crypto/md5 computation, which panics in FIPS 140-only
mode, with a generated table. Keeps the rolling hash in locals in the per-byte
loop, fixes single-subtest leaks in tests, and wraps option errors in
arrow.ErrInvalid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…name error test, document CDC options

Adds race-detector build-tagged constants, a reduced type matrix for
TestCDCSingleRowGroup under -race, documents NormLevel and the Go-specific
CDC behaviors, and drops the STRING condition on the plainPages assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e chunking loop

Keep the rolling state in a by-value struct local to Chunks, merge the three
level loops into one, and roll 1/2/4/8-byte values as unrolled words inline.
Chunk boundaries are unchanged (pinned by TestChunksStableBoundaries).

BenchmarkChunks, 1Mi levels, DefaultCDCOptions, Apple M4 (median ms):
  required/int64      5.7 -> 3.5
  nullable/int64      7.8 -> 4.1
  nested/list<int32> 10.3 -> 5.2
  required/string     7.5 -> 8.3
The per-value byte path (strings/binary) got about 0.8 ns/value slower. End to
end, pqarrow writes of 1Mi rows with dictionary on: the 7-column primitive CDC
write went from 39.6 to 31.3 ms, the string CDC write from 33.5 to 34.7 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- writeArrowToColumn: drop the unreachable len(chunks)==0 branch and
  resolve FlushCurrentPage once before the chunk loop.
- Have the gear table generator emit numGearTables and remove
  gearhash.go; the table values are unchanged.
- Document why newValueSource matches binary-like arrays by concrete
  type (an interface case is about 10% slower on strings).
- Tests: table-driven edit scenarios and mask cases, shared input and
  reader helpers, array.NewTableFromSlice for table assembly, and the
  multi row group tables generated once. Test names and assertions are
  unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Chunks returns arrow.ErrInvalid when the levels can't cover the values,
  instead of returning no chunks and silently dropping the data.
- Hash fixed-width values as their little-endian bytes so chunk
  boundaries don't depend on the host's byte order.
- Skip the dictionary comparison for slices sharing the preserved
  dictionary's data, so CDC chunks don't compare it once per chunk.
- Release each chunk's value slice in a deferred call.
- WriteArrowToColumn returns arrow.ErrNotImplemented when CDC is enabled.
- Add FlushCurrentPage to file.ColumnChunkWriter.
- Compile the chunk validation only with the assert build tag.
- Wrap the unsupported-type error with arrow.ErrNotImplemented.
- Document that chunking state spans row groups, as in Arrow C++ 27.
- Tests: test through the public API, pin the gear tables to the C++
  header, and cover decimals in the word/byte equivalence test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share the round-trip and page-size helpers across the CDC tests, use
existing arrow helpers (NewTableReader, ListLikeType, TotalArraySize,
mockpagewriter) instead of hand-written equivalents, and inline helpers
used once. Validate chunks against the real value count in assert
builds and convert decimal words to little-endian on every host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zeroshade

Copy link
Copy Markdown
Member

Hey @kszucs! This has been on my radar and list of stuff to do for a while, ever since I saw your blog post about it. So I'm super excited to take a look at this. Thanks much!

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a careful port. I compared Chunks against the C++ chunker. The min-size skip, the 8-match/max-size cut, cutting before the triggering value, hashing levels as 2 little-endian bytes, never resetting the hash, and cutting nested columns only at rep == 0 all match. Null slots hash only their definition level, and value slots line up with what pqarrow slices per chunk. The FlushCurrentPage early return is safe because numBufferedValues counts levels, so all-null pages are still written. The dictionary shortcut only skips the comparison when both sides share the same *Data.

Locally (Apple M4, Go 1.27.1), go test -tags assert passes for parquet/internal/cdc, parquet/pqarrow and parquet/file, -race passes for cdc and pqarrow, and go vet passes for s390x.

API break. Adding FlushCurrentPage() error to the exported file.ColumnChunkWriter interface breaks implementations outside this module in a v18 minor release. Since every column writer here already has the method, writeArrowToColumn could use an optional interface instead, e.g. cw.(interface{ FlushCurrentPage() error }), and return an error if it's missing. That keeps the interface unchanged. The other two listed changes (no empty flushes, WriteArrowToColumn rejecting CDC) seem fine.

Non-blocking:

  • Fixed-width values of 8 bytes take the native-word path. For multi-field types such as DAY_TIME intervals that gives a different byte order on big-endian hosts than on little-endian ones, so the byte-order-independence claim doesn't hold there. pqarrow doesn't write those types today, so a comment, or limiting the word path to single-field types, would do.
  • The boolean path builds []byte{1} / []byte{0} per value; two package-level arrays would avoid it.
  • A small fixture written by PyArrow with CDC would pin cross-implementation compatibility, which the description notes isn't covered yet.
  • At ~5.3k lines (most of it generated tables and tests) this is reviewable, but splitting the file package change into its own commit/PR would make history easier to follow.

I'm also looking at the write overhead with CDC enabled (about 3x for int64 and 2x for strings in BenchmarkWriteContentDefinedChunking here) and will follow up with any profiling results.

@zeroshade

Copy link
Copy Markdown
Member

@kszucs following up on the write overhead with CDC enabled. I profiled it and prototyped some changes to chunker.go that keep the page boundaries bit-identical. TestChunksStableBoundaries and the full pqarrow CDC suite pass with -tags assert and -tags assert,noasm, and -race passes for cdc.

Where the time goes (pprof, Apple M4): in BenchmarkChunks almost everything is Chunks. For required int64 roughly 35% is hashing and ~55% is per-value overhead of the general loop (level checks, width dispatch, needNewChunk, offset bookkeeping). Strings also pay a non-inlinable closure call per value.

Changes that helped:

  1. Skip the min-size window instead of walking it. Values that only fill MinChunkSize are never hashed and can't end a chunk, so they only need their sizes added. Required fixed-width columns skip arithmetically, required binary/string columns binary-search the offsets, and nullable/nested fixed-width columns use a tight size-only loop.
  2. Specialized loops for required 8-byte values and required binary/string values, reading the data and offsets buffers directly with no level handling or per-value closure.
  3. Split the hash chain per word. For a word, the hash after byte j is hash<<(j+1) + p_j, where p_j rolls the word's own bytes from zero and doesn't depend on hash. The loop-carried chain becomes one shift-add per word instead of eight. hash&mask == 0 is checked as (hash>>shift) - 1 having its top bit set, OR-ed over the word, which avoids a compare per byte. On its own this was only ~6%; most of the gain is from 1 and 2.

BenchmarkChunks:

Input PR Prototype Speedup
required int64 2335 MB/s 4358 MB/s 1.87x
nullable int64 1886 MB/s 2140 MB/s 1.13x
nested list 650 MB/s 784 MB/s 1.21x
required string 2059 MB/s 3391 MB/s 1.65x

BenchmarkWriteContentDefinedChunking with CDC on (median of 5 interleaved runs):

Column PR Prototype Change
int64 5.13 ms 3.52 ms -31%
string 21.65 ms 18.63 ms -14%

SIMD: I don't think it's worth it here. The gear hash has to stay as is for compatibility with C++ and arrow-rs, and vectorizing it needs 64-bit gathers. The FAST'25 VectorCDC paper (Udayashankar et al.) measured only ~1.2x for AVX-512 Gear and no gain for FastCDC, because splitting hashing from boundary detection gives up min-size skipping. NEON has no 64-bit gather, so arm64 wouldn't benefit at all.

Other ideas, not prototyped:

  • For strings, roll the contiguous value bytes as one stream and only stop at matches (~1 in 32k bytes with the defaults) or the max size, which would remove the remaining per-value cost.
  • The same specialization for required 4-byte and flat nullable 8-byte columns. Nullable matters since Arrow fields default to nullable.
  • Not CDC-specific: PooledBufferWriter.Reset and poolBuffer.Release memset the whole buffer capacity on every page flush. That was ~13% of the string write profile, and CDC makes it more visible because it produces more pages.

None of this blocks the PR. Happy to share the patch (~200 lines, chunker.go only) if you'd like to fold it in here, or I can send it as a follow-up after this merges.

@kszucs

kszucs commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Thanks @zeroshade for reviewing and optimizing! I'm busy with the java/cpp/rust implementations but circling back as soon as I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants