Repository navigation
Conversation
parquet-mr zero-pads every encrypted thrift module to its buffer size, so the decrypted page header is the thrift struct followed by zero bytes. readPageHeader worked out how far to advance from the amount of thrift it parsed. That stopped partway through the encrypted module, and the next read took its page data from inside it. Advance by the length of the framed ciphertext instead. external_key_material_java.parquet.encrypted from parquet-testing now reads back its 100 rows instead of failing in the first data page.
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, this is a good catch with a clear root cause. I confirmed it: parquet-java's Util.write encrypts TMemoryBuffer.getArray(), so every module is padded to at least 100 bytes. The fixture says parquet-mr 1.12.0, while the other encrypted fixtures are all from parquet-cpp-arrow, which is why nothing caught it. Skipping the whole frame also matches what C++ does (DeserializeMessage takes the bytes read from the full decrypted length). Both new tests fail on main without the fix and pass with it.
One thing before this lands, and a nit, both inline:
TestDecryptParquetMrExternalKeyMaterialshould check the values as well as the shape, like the C++ReadParquetMRExternalKeyMaterialFiletest does. Otherwise a reader that lands on the wrong bytes but still decodes 100 rows would pass.- Nit: explain the 76 padding bytes.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.
…padding TestDecryptParquetMrExternalKeyMaterial now checks every value the way the C++ ReadParquetMRExternalKeyMaterialFile test does (column 0 is the row index, column 1 is prefixes[row % 10] followed by the row index), so a reader that lands on the wrong bytes but still decodes 100 rows fails. The padded-header test derives its padding from parquet-mr's 100-byte INIT_MEM_ALLOC_ENCR_BUFFER instead of a bare 76.
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, both addressed. The fixture test now checks every value the way the C++ ReadParquetMRExternalKeyMaterialFile test does, and the padding is derived from parquet-mr's 100-byte initial buffer with an explanation. LGTM.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.
### Rationale for this change #566 reports encryption and decryption failures between pyarrow and arrow-go. The direct-key half no longer reproduces on main: reading pyarrow-written files in Go works for uniform AES-GCM, AES-GCM-CTR, AAD prefix and plaintext footer, and pyarrow reads an arrow-go-written file. Checked against pyarrow 26; I suspect #596/#1390/#1412 fixed this but haven't bisected. Nothing tests these paths in either direction today, so it could regress unnoticed. What still fails is KMS. pyarrow's `CryptoFactory` writes PKMT1 key material with a wrapped DEK, arrow-go has no parser or unwrapping for it, so footer decryption fails without a custom retriever. That gap stays tracked in #566; per review the skip-only KMS test was left out of this PR and belongs with the PKMT1 follow-up. ### What changes are included in this PR? `encryption_interop_pyarrow.py` generates the encrypted files and reads a Go-written one; it is embedded in the test binary and run through `python3`. `encryption_interop_pyarrow_test.go` holds the tests. They skip when `python3` or `pyarrow` isn't installed, so the default test run is unchanged. `plainfooter_direct` also asserts the footer magic stays `PAR1` (an encrypted footer would end in `PARE1`) so the case checks what its name says. CI: `ci/scripts/cgo_python_test.sh`, which runs in the `docker-cgo-python` job (the only one with pyarrow installed), now runs `go test -run Pyarrow` on the package with `PARQUET_TEST_REQUIRE_PYARROW=1`, which turns the "pyarrow not available" skip into a failure so an image change can't silently disable the tests. Since the install is unpinned, the tests follow the current pyarrow release. ### Are these changes tested? The tests are the change. Against pyarrow 26 locally: the four direct-key configs pass and pyarrow reads the Go-written file back correctly. Without python3 or pyarrow everything skips. ### Are there any user-facing changes? No, tests only.
Rationale for this change
arrow-go cannot read a Parquet file that parquet-mr (Spark, Hive, Iceberg, anything on parquet-java) wrote with modular encryption.
file.OpenParquetFilesucceeds and the first page read fails:I hit it with
data/external_key_material_java.parquet.encryptedfrom parquet-testing.parquet-mr zero-pads every encrypted thrift module to the size of its buffer:
Util.javaencrypts the whole backing array of itsTMemoryBuffer, not only the bytes written. So an encrypted page header is the thrift struct followed by zeros, all inside one length-prefixed module.readPageHeader(parquet/file/page_reader.go:796) decrypts the module, parses the thrift, and then advances the stream by the amount of thrift it consumed plus the cipher overhead (line 837). That is only right when nothing follows the struct. With padding it stops part way through the module, and the next read takes ciphertext for page data. Files from arrow-go and from C++ (the ones in parquet-testing) are not padded, which is why the existing tests never saw it.What changes are included in this PR?
For an encrypted header,
readPageHeadernow skips the whole framed ciphertext (the 4-byte length prefix plus the module, whichencryption.FramedCiphertextalready returns). The unencrypted path is unchanged.Are these changes tested?
Yes. Both new tests fail on main and pass with the change.
TestDecryptParquetMrExternalKeyMaterial(parquet/encryption_read_config_test.go) reads the parquet-testing file above and expects 100 rows and 2 columns. That file uses external key material with double wrapping and arrow-go has no key toolkit, so the test carries a small key retriever that unwraps the keys with the same master keys the C++ tests use (kf,kc1,kc2). On main it fails with the error above.TestReadEncryptedPageHeaderWithTrailingPadding(parquet/file/page_reader_internal_test.go) encrypts a page header whose plaintext is the thrift struct plus 76 zero bytes, puts page data after it, and checks thatreadPageHeaderleaves the stream at the page data. On main it is left inside the module.Commands, with
PARQUET_TEST_DATA=$PWD/parquet-testing/data:go test ./parquet -run 'TestFileEncryptionDecryption|TestDecryptParquetMrExternalKeyMaterial' -count=1: ok (the existing C++-written configurations still pass)go test ./parquet/file -run 'TestReadEncryptedPageHeaderWithTrailingPadding|TestPageReader' -count=1: okgo vet ./parquet ./parquet/fileandgofmt -lon the changed files: cleanThe fixture is the one in the parquet-testing submodule this repo already pins. Its README says it was encrypted with parquet-mr using external key material and the in-memory KMS client. No fixture files were added.
Are there any user-facing changes?
Yes. Encrypted files written by parquet-mr / parquet-java can be read. Nothing changes for files written by arrow-go or C++.
Related to #566. That issue reports that cross-library encryption does not work in general; this covers the parquet-mr to Go read path only.
AI/LLM disclosure