Skip to content

fix(parquet/file): read encrypted page headers written by parquet-mr - #1390

Merged
zeroshade merged 2 commits into
apache:mainfrom
sankalpsthakur:fix/parquet-encrypted-page-header-padding
Oct 21, 2026
Merged

zeroshade merged 2 commits into
apache:mainfrom
sankalpsthakur:fix/parquet-encrypted-page-header-padding

Conversation

@sankalpsthakur

Copy link
Copy Markdown
Contributor

Rationale for this change

arrow-go cannot read a Parquet file that parquet-mr (Spark, Hive, Iceberg, anything on parquet-java) wrote with modular encryption. file.OpenParquetFile succeeds and the first page read fails:

decrypting data page: encrypted payload length prefix is 4204214195, available length is 40: unexpected EOF

I hit it with data/external_key_material_java.parquet.encrypted from parquet-testing.

parquet-mr zero-pads every encrypted thrift module to the size of its buffer: Util.java encrypts the whole backing array of its TMemoryBuffer, not only the bytes written. So an encrypted page header is the thrift struct followed by zeros, all inside one length-prefixed module. readPageHeader (parquet/file/page_reader.go:796) decrypts the module, parses the thrift, and then advances the stream by the amount of thrift it consumed plus the cipher overhead (line 837). That is only right when nothing follows the struct. With padding it stops part way through the module, and the next read takes ciphertext for page data. Files from arrow-go and from C++ (the ones in parquet-testing) are not padded, which is why the existing tests never saw it.

What changes are included in this PR?

For an encrypted header, readPageHeader now skips the whole framed ciphertext (the 4-byte length prefix plus the module, which encryption.FramedCiphertext already returns). The unencrypted path is unchanged.

Are these changes tested?

Yes. Both new tests fail on main and pass with the change.

  • TestDecryptParquetMrExternalKeyMaterial (parquet/encryption_read_config_test.go) reads the parquet-testing file above and expects 100 rows and 2 columns. That file uses external key material with double wrapping and arrow-go has no key toolkit, so the test carries a small key retriever that unwraps the keys with the same master keys the C++ tests use (kf, kc1, kc2). On main it fails with the error above.
  • TestReadEncryptedPageHeaderWithTrailingPadding (parquet/file/page_reader_internal_test.go) encrypts a page header whose plaintext is the thrift struct plus 76 zero bytes, puts page data after it, and checks that readPageHeader leaves the stream at the page data. On main it is left inside the module.

Commands, with PARQUET_TEST_DATA=$PWD/parquet-testing/data:

  • go test ./parquet -run 'TestFileEncryptionDecryption|TestDecryptParquetMrExternalKeyMaterial' -count=1: ok (the existing C++-written configurations still pass)
  • go test ./parquet/file -run 'TestReadEncryptedPageHeaderWithTrailingPadding|TestPageReader' -count=1: ok
  • go vet ./parquet ./parquet/file and gofmt -l on the changed files: clean

The fixture is the one in the parquet-testing submodule this repo already pins. Its README says it was encrypted with parquet-mr using external key material and the in-memory KMS client. No fixture files were added.

Are there any user-facing changes?

Yes. Encrypted files written by parquet-mr / parquet-java can be read. Nothing changes for files written by arrow-go or C++.

Related to #566. That issue reports that cross-library encryption does not work in general; this covers the parquet-mr to Go read path only.

AI/LLM disclosure

  • AI coding tools (Claude Code) were used to find and write this fix, the two tests and this description.
  • The failure was reproduced on main, the change was reviewed, and the commands above were run locally before submitting.

parquet-mr zero-pads every encrypted thrift module to its buffer size, so
the decrypted page header is the thrift struct followed by zero bytes.
readPageHeader worked out how far to advance from the amount of thrift it
parsed. That stopped partway through the encrypted module, and the next
read took its page data from inside it. Advance by the length of the
framed ciphertext instead.

external_key_material_java.parquet.encrypted from parquet-testing now
reads back its 100 rows instead of failing in the first data page.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a good catch with a clear root cause. I confirmed it: parquet-java's Util.write encrypts TMemoryBuffer.getArray(), so every module is padded to at least 100 bytes. The fixture says parquet-mr 1.12.0, while the other encrypted fixtures are all from parquet-cpp-arrow, which is why nothing caught it. Skipping the whole frame also matches what C++ does (DeserializeMessage takes the bytes read from the full decrypted length). Both new tests fail on main without the fix and pass with it.

One thing before this lands, and a nit, both inline:

  • TestDecryptParquetMrExternalKeyMaterial should check the values as well as the shape, like the C++ ReadParquetMRExternalKeyMaterialFile test does. Otherwise a reader that lands on the wrong bytes but still decodes 100 rows would pass.
  • Nit: explain the 76 padding bytes.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.

Comment thread parquet/encryption_read_config_test.go Resolved
…padding

TestDecryptParquetMrExternalKeyMaterial now checks every value the way the
C++ ReadParquetMRExternalKeyMaterialFile test does (column 0 is the row index,
column 1 is prefixes[row % 10] followed by the row index), so a reader that
lands on the wrong bytes but still decodes 100 rows fails.

The padded-header test derives its padding from parquet-mr's 100-byte
INIT_MEM_ALLOC_ENCR_BUFFER instead of a bare 76.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, both addressed. The fixture test now checks every value the way the C++ ReadParquetMRExternalKeyMaterialFile test does, and the padding is derived from parquet-mr's 100-byte initial buffer with an explanation. LGTM.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.

@zeroshade
zeroshade merged commit de0c94e into apache:main Oct 21, 2026
23 checks passed
zeroshade pushed a commit that referenced this pull request Nov 4, 2026
### Rationale for this change

#566 reports encryption and decryption failures between pyarrow and
arrow-go. The direct-key half no longer reproduces on main: reading
pyarrow-written files in Go works for uniform AES-GCM, AES-GCM-CTR, AAD
prefix and plaintext footer, and pyarrow reads an arrow-go-written file.
Checked against pyarrow 26; I suspect #596/#1390/#1412 fixed this but
haven't bisected. Nothing tests these paths in either direction today,
so it could regress unnoticed.

What still fails is KMS. pyarrow's `CryptoFactory` writes PKMT1 key
material with a wrapped DEK, arrow-go has no parser or unwrapping for
it, so footer decryption fails without a custom retriever. That gap
stays tracked in #566; per review the skip-only KMS test was left out of
this PR and belongs with the PKMT1 follow-up.

### What changes are included in this PR?

`encryption_interop_pyarrow.py` generates the encrypted files and reads
a Go-written one; it is embedded in the test binary and run through
`python3`. `encryption_interop_pyarrow_test.go` holds the tests. They
skip when `python3` or `pyarrow` isn't installed, so the default test
run is unchanged. `plainfooter_direct` also asserts the footer magic
stays `PAR1` (an encrypted footer would end in `PARE1`) so the case
checks what its name says.

CI: `ci/scripts/cgo_python_test.sh`, which runs in the
`docker-cgo-python` job (the only one with pyarrow installed), now runs
`go test -run Pyarrow` on the package with
`PARQUET_TEST_REQUIRE_PYARROW=1`, which turns the "pyarrow not
available" skip into a failure so an image change can't silently disable
the tests. Since the install is unpinned, the tests follow the current
pyarrow release.

### Are these changes tested?

The tests are the change. Against pyarrow 26 locally: the four
direct-key configs pass and pyarrow reads the Go-written file back
correctly. Without python3 or pyarrow everything skips.

### Are there any user-facing changes?

No, tests only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants