Repository navigation
fix(parquet/schema): ignore logical types this version does not know - #1649
Open
sankalpsthakur wants to merge 1 commit into
Open
sankalpsthakur wants to merge 1 commit into
sankalpsthakur wants to merge 1 commit into
Conversation
A file written with a logical type added after this version made file.OpenParquetFile panic with "invalid logical type": the thrift decoder skips the unknown union member and getLogicalType had no case for the empty union it leaves behind. LogicalTypes.md asks readers to ignore the annotation and the column order of such a column and use its physical type only, so it now reads as having no logical type and its sort order is UNKNOWN, which keeps min/max statistics from being trusted (as Arrow C++ does). The converted_type is left as it is. unknown-logical-type.parquet from parquet-testing is read as a string column and a binary column. The branch for a nil union is dropped: it could never run, because IsSetSTRING dereferenced the nil union first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A Parquet file that uses a logical type added after this version of arrow-go makes
file.OpenParquetFilepanic withinvalid logical type. The thrift decoder skips the union member it does not know and leaves an emptyLogicalTypeunion behind.getLogicalType(parquet/schema/logical_types.go) has a case for every member, and nothing for the empty one, so it reachespanic("invalid logical type"). The oldcase l == nilcould never run, becausel.IsSetSTRING()dereferences the nil union first.The format says what a reader should do (parquet-format
LogicalTypes.md, "Unsupported Logical Types"): "readers should ignore both the logical type annotation and column order for that column. Only the physical type information should be used to process the column's data." Arrow C++ does this since apache/arrow#41765. arrow-go is one of the implementations listed without a fix in apache/parquet-format#599.What changes are included in this PR?
getLogicalTypereturnsNoLogicalTypefor a nil union and for an empty one. The deadcase l == nilbranch is removed.PrimitiveNodeFromThriftmarks such a node, andColumn.SortOrder()returnsSortUNKNOWNfor it. Without this the column becameNoLogicalType, which maps to the default order of the physical type (unsigned for BYTE_ARRAY), and the file's min/max statistics were trusted even though the unknown type may define another order. Arrow C++ makes the sort order UNKNOWN as well.converted_type: behaviour is unchanged. If the element also has aconverted_typenext to the unknown logical type (for example UTF8 or LIST), it is dropped with the annotation, and the node reads as plain physical type or plain group. A newer logical type has no converted equivalent, so the case is rare. I kept the PR to the logical type path, like #41765, and can fall back to the converted type in a follow-up if you prefer that.getLogicalTypestill panics for a TIME or TIMESTAMP whoseTimeUnitunion is empty, and an unrecognized logical/physical type combination still returns an error fromNewPrimitiveNodeLogical. Writing such a file back drops the unknown annotation, becauseNoLogicalTypeis not serialized.Are these changes tested?
Yes.
TestUnknownThriftLogicalType(parquet/schema/unknown_logical_type_test.go) covers a primitive and a group element with and without aconverted_type, and assertsSortUNKNOWNfor the primitive column. It fails without the sort order change (expected 2, actual 1).TestReadUnknownLogicalType(parquet/pqarrow/unknown_logical_type_test.go) readsunknown-logical-type.parquetfrom parquet-testing (a string column and a binary column) with a checked allocator.TestInvalidConvertedTypeInDeserializeused to expect the panic and now expects the node to read as having no logical type.Run:
PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema ./parquet/pqarrow -run 'Unknown|Invalid' -count=1: okPARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema -count=1: okgo vet ./parquet/schema ./parquet/pqarrowandgofmt -l: cleanAre there any user-facing changes?
Yes. Files that use a logical type from a newer writer can be opened and read by their physical type instead of panicking. Min/max statistics of such a column are not used for pruning. There is no API change.
AI/LLM disclosure