You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Format] Specify FIXED_SIZE_LIST Logical type #430
Arrow recently introduced FixedShapeTensor and VariableShapeTensor canonical extension types that use FixedSizeList and StructArray(List, FixedSizeList) as storage respectfully. These are targeted at machine learning and scientific applications that deal with large datasets and would benefit from using Parquet as on disk storage.
However currently FixedSizeList is stored as List in Parquet which adds significant conversion overhead when reading and writing as discussed here. It would therefore be beneficial to introduce a FIXED_SIZE_LIST logical type to Parquet.
Activity
apacheGH-430: Add FIXED_SIZE_LIST logical type
apacheGH-430: Add FIXED_SIZE_LIST logical type
apacheGH-430: Support LIST and FLBA representations
apacheGH-430: Add LIST-based VECTOR logical type
apacheGH-430: Add LIST-based VECTOR logical type
apacheGH-430: Add LIST-based VECTOR logical type
apacheGH-430: Add LIST-based VECTOR logical type
Add extension types with parquet.vector as the first canonical extension
Add extension types with parquet.vector as the first canonical extension
Add parquet.vector canonical extension type
apacheGH-430: Add LIST-based VECTOR logical type
GH-430: [Format] Add FINITE_VECTOR logical type with LIST storage (#624)