Skip to content

fix(parquet/schema): ignore logical types this version does not know - #1649

Open
sankalpsthakur wants to merge 1 commit into
apache:mainfrom
sankalpsthakur:fix/parquet-unknown-logical-type-v2
Open

sankalpsthakur wants to merge 1 commit into
apache:mainfrom
sankalpsthakur:fix/parquet-unknown-logical-type-v2

Conversation

@sankalpsthakur

Copy link
Copy Markdown
Contributor

Rationale for this change

A Parquet file that uses a logical type added after this version of arrow-go makes file.OpenParquetFile panic with invalid logical type. The thrift decoder skips the union member it does not know and leaves an empty LogicalType union behind. getLogicalType (parquet/schema/logical_types.go) has a case for every member, and nothing for the empty one, so it reaches panic("invalid logical type"). The old case l == nil could never run, because l.IsSetSTRING() dereferences the nil union first.

The format says what a reader should do (parquet-format LogicalTypes.md, "Unsupported Logical Types"): "readers should ignore both the logical type annotation and column order for that column. Only the physical type information should be used to process the column's data." Arrow C++ does this since apache/arrow#41765. arrow-go is one of the implementations listed without a fix in apache/parquet-format#599.

What changes are included in this PR?

  • getLogicalType returns NoLogicalType for a nil union and for an empty one. The dead case l == nil branch is removed.
  • Logical type: a primitive column with an unknown annotation reads as having no logical type, so it is processed by its physical type.
  • Column order: PrimitiveNodeFromThrift marks such a node, and Column.SortOrder() returns SortUNKNOWN for it. Without this the column became NoLogicalType, which maps to the default order of the physical type (unsigned for BYTE_ARRAY), and the file's min/max statistics were trusted even though the unknown type may define another order. Arrow C++ makes the sort order UNKNOWN as well.
  • A group node with an unknown annotation is built with no logical type.
  • converted_type: behaviour is unchanged. If the element also has a converted_type next to the unknown logical type (for example UTF8 or LIST), it is dropped with the annotation, and the node reads as plain physical type or plain group. A newer logical type has no converted equivalent, so the case is rare. I kept the PR to the logical type path, like #41765, and can fall back to the converted type in a follow-up if you prefer that.
  • Out of scope: getLogicalType still panics for a TIME or TIMESTAMP whose TimeUnit union is empty, and an unrecognized logical/physical type combination still returns an error from NewPrimitiveNodeLogical. Writing such a file back drops the unknown annotation, because NoLogicalType is not serialized.

Are these changes tested?

Yes.

  • TestUnknownThriftLogicalType (parquet/schema/unknown_logical_type_test.go) covers a primitive and a group element with and without a converted_type, and asserts SortUNKNOWN for the primitive column. It fails without the sort order change (expected 2, actual 1).
  • TestReadUnknownLogicalType (parquet/pqarrow/unknown_logical_type_test.go) reads unknown-logical-type.parquet from parquet-testing (a string column and a binary column) with a checked allocator.
  • TestInvalidConvertedTypeInDeserialize used to expect the panic and now expects the node to read as having no logical type.

Run:

  • PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema ./parquet/pqarrow -run 'Unknown|Invalid' -count=1: ok
  • PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema -count=1: ok
  • go vet ./parquet/schema ./parquet/pqarrow and gofmt -l: clean

Are there any user-facing changes?

Yes. Files that use a logical type from a newer writer can be opened and read by their physical type instead of panicking. Min/max statistics of such a column are not used for pruning. There is no API change.

AI/LLM disclosure

  • AI coding tools were used to help draft or modify code and this PR description.
  • I reviewed the complete change, understand the reasoning, and ran the reported local tests before submitting.

A file written with a logical type added after this version made
file.OpenParquetFile panic with "invalid logical type": the thrift decoder
skips the unknown union member and getLogicalType had no case for the
empty union it leaves behind. LogicalTypes.md asks readers to ignore the
annotation and the column order of such a column and use its physical type
only, so it now reads as having no logical type and its sort order is
UNKNOWN, which keeps min/max statistics from being trusted (as Arrow C++
does). The converted_type is left as it is. unknown-logical-type.parquet
from parquet-testing is read as a string column and a binary column.

The branch for a nil union is dropped: it could never run, because
IsSetSTRING dereferenced the nil union first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant