Repository navigation
Conversation
This diff adds support for sorting by nested fields. The primary motivation for this change is enabling support for sorting by nested fields in iceberg-go, which is supported by the Python and Java implementations. * `compute.SortKey` now has a `ColumnPath` field that can be used to fully address nested struct fields. `SortKey.ColumnIndex` is preserved for backwards compat/ergo. `ColumnPath` takes precedence over `ColumnIndex` if both are specified. * `sort_indices` now walks the `ColumnPath` if it is specified. * I've added some new benchmarks for `compute.SortRecordBatch`. * **Note** `maskedStructField` is basically a copy of `Struct.newStructFieldWithParentValidityMask`. I opted to copy this rather than exporting a increasing the surface area of the public API. Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
zeroshade
left a comment
There was a problem hiding this comment.
Requesting changes: nested-path sorting returns wrong order on sliced input, and the new tests don't exercise parent-validity masking.
- Sliced input sorts incorrectly (inline on
vector_sort.go:154). - Masking is untested: the tests still pass with
maskedStructFieldforced to return the raw child (inline on the test). SortKeyno longer supports==: it has been public since v18.6.0 andColumnPath []intmakes it non-comparable (inline onColumnPath). This needs a deliberate API decision; a mailing-list thread seems worthwhile, as you offered. Arrow C++SortKeyuses aFieldReftarget, which may be the more consistent shape.
Minor: the SortRecordBatch/SortTable docs still only mention ColumnIndex, and SortIndicesArray/SortIndicesChunked silently ignore ColumnPath.
FYI, not caused by this PR: kernels.SortIndices never releases its tmpBuf scratch allocations, so a CheckedAllocator AssertSize(t, 0) on any nullable sort key reports a 64-byte leak on main. I'll handle that separately.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.
| // Normalize to a zero offset so maskedNullBitmapBytes (indexed from | ||
| // fieldOffset above) lines up with the sliced buffers. | ||
| sliced := array.NewSliceData(field.Data(), 0, int64(field.Len())) | ||
| defer sliced.Release() | ||
|
|
||
| origBufs := sliced.Buffers() | ||
| bufs := make([]*memory.Buffer, len(origBufs)) | ||
| copy(bufs, origBufs) | ||
| bufs[0] = memory.NewBufferBytes(maskedNullBitmapBytes) | ||
|
|
||
| data := array.NewData(sliced.DataType(), sliced.Len(), bufs, sliced.Children(), array.UnknownNullCount, 0) |
There was a problem hiding this comment.
NewSliceData(d, 0, n) keeps d.Offset() and shares the buffers; it does not reset the offset to zero. Passing 0 here makes the new array read values from index 0 and misaligns the bitmap, which was built at fieldOffset+i.
Repro: ids [100,101,1,2,3,4], ranks [-5,-4,30,null,10,20], then batch.NewSlice(2,6) and sort by {0,1,0}. The result is [1 2 3 4]; the expected order is [3 4 1 2]. Using sliced.Offset() fixes it. You could also skip the NewSliceData round-trip and build from field.Data() directly. Please add a sliced batch/chunk test.
|
|
||
| // Row 1: id=2, info=null. StructBuilder.Append(false) auto-appends | ||
| // null to info's children (rank), so rankBldr is not touched here; | ||
| // rank's underlying storage slot still ends up non-null-looking at | ||
| // the physical level, which is exactly what parent-validity masking | ||
| // must override. | ||
| bldr.Append(true) | ||
| idBldr.Append(2) | ||
| infoBldr.Append(false) |
There was a problem hiding this comment.
StructBuilder.Append(false) calls AppendNull() on every child (array/struct.go:395-398), so rank is already physically null here and the comment is inaccurate. As a result, this test and the multi-chunk table test pass even when maskedStructField just returns st.Field(i) unmasked (I checked).
Please build a case where the parent is null but the child slot is valid, e.g. with array.NewStructArrayWithNulls or NewStructData and a hand-built bitmap.
| // the descendant's own physical validity bitmap. Only struct nesting is | ||
| // supported; a path element pointing into a list, map, or union column | ||
| // is an error. | ||
| ColumnPath []int |
There was a problem hiding this comment.
Adding a slice field makes SortKey non-comparable, so existing key == other / map-key usage stops compiling. SortKey has been public since v18.6.0, so this is a breaking change in a minor release. It needs a deliberate call. Arrow C++ SortKey takes a FieldRef target, which may be the more consistent shape.
| leafType = resolved.DataType() | ||
| } | ||
| if leafType == nil { | ||
| leafType = chunked.DataType() |
There was a problem hiding this comment.
With zero chunks, the path is never validated and leafType falls back to the struct type. Validating the path once against the column type (compute.FieldPath.GetFieldFromType) before the per-chunk loop would give the correct leaf type and catch bad paths even on empty tables.
| } | ||
| idx := path[0] | ||
| if idx < 0 || idx >= st.NumField() { | ||
| return nil, fmt.Errorf("%w: sort key struct field index %d out of range", arrow.ErrIndex, idx) |
There was a problem hiding this comment.
nit: an out-of-range top-level index returns ErrInvalid, but an out-of-range nested index returns ErrIndex. Pick one.
* adds additional tests to repro bugs found in review * fixes offset bug in maskedStructField * makes resolveSortColumnPath return ErrInvalid in both err cases * evaluate leaf type once rather than per chunk
|
I appreciate the review @zeroshade . I didn't consider direct comparability of |
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. 1ea3b385 fixes every code issue from the last round. I checked each one:
- Sliced input: fixed. The new sliced batch and chunk tests pass.
- Masking: fixed. Forcing
maskedStructFieldto return the child unmasked now makes 4 of the nested tests fail, so they catch it. - Zero-chunk path validation and leaf type: fixed.
ErrIndex/ErrInvalid: fixed.
Still open:
SortKeycomparability (inline). Blocked on the mailing-list thread.SortIndicesArray/SortIndicesChunkedignoreColumnPath. With a struct array andColumnPath: {0,1}, they returnnot implemented: sorting not supported for type STRUCT, which is misleading. Either resolve the path for these inputs or reject a non-emptyColumnPathwithErrInvalidand document it. (The lines that resetkey.ColumnIndex = 0aren't in the diff, so this can't go inline.)- Docs: the
SortRecordBatch/SortTableexamples only showColumnIndex. Please add aColumnPathexample.
| // the descendant's own physical validity bitmap. Only struct nesting is | ||
| // supported; a path element pointing into a list, map, or union column | ||
| // is an error. | ||
| ColumnPath []int |
There was a problem hiding this comment.
Still open: []int makes SortKey non-comparable. Holding this until the mailing-list thread settles the shape (slice vs. a C++-style FieldRef).
Today writing to a table whose default sort order sources a key from a nested field fails, because the writer sorts each batch with `compute.SortRecordBatch` and arrow-go's `compute.SortKey` can only address top-level columns. This diff adds support for nested sort keys by falling back to the row comparator from sort compaction (apache#2345) when a key is nested in a struct. The comparator resolves each key to a struct path and treats a null ancestor struct as a null key. Sort orders whose keys are all top-level columns still use arrow-go, so that we pick up any improvements made there (e.g. the radix sort added in v18.9.0). The fallback is intended to only stick around until arrow-go can provide support for sorting by nested fields. Notes: * This implementation is entirely internal to the table package so we should be able to swap it out later. * I am working on getting support for nested sort keys added to arrow-go (see apache/arrow-go#1337). * Bonus: compaction operations now support nested sort keys. * Sources under lists or maps are rejected w/ `ErrInvalidArgument` as they are not allowed by the Iceberg spec. * We now set `FloatTotalOrder` on arrow's sort keys so top-level float keys match the Iceberg spec's float order, same as the comparator. Previously, arrow placed NaN next to nulls and treated -0 as equal to 0. * Adds new benchmarks. * I benchmarked appending 1M rows against main: top-level keys run as fast as before, and a nested float64 key takes ~699 ms vs ~402 ms unsorted. Total order also speeds up float keys: a 64K-row float64 batch w/o nulls sorts in ~2.0 ms instead of ~18.5 ms. Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
This diff adds support for writing to tables that define sort keys from nested struct fields. Previously attempting to write to a table with such a sort order defined would error. With this diff: * Tables whose sort keys are all top-level fields use arrow-go directly, same as before. * Tables with any sort key on a nested field fall back to the row comparator used by sort compaction (introduced with apache#2345). * This implementation is entirely internal to the table package so we should be able to swap it out later. * Sources under lists or maps are rejected w/ `ErrInvalidArgument` as they are not allowed by the Iceberg spec. * We now set `FloatTotalOrder` on arrow's sort keys so top-level float keys match the Iceberg spec's float order, same as the comparator. Previously, arrow placed NaN next to nulls and treated -0 as equal to 0, so tables w/ a top-level float sort key now write rows w/ NaN or -0 keys in a different order. This is also faster: a 64K-row float64 batch w/o nulls sorts in ~2.0 ms instead of ~18.5 ms. Other Notes: * Long term we should add support to arrow-go. I am working on that (see apache/arrow-go#1337) and plan to follow up w/ a change removing this logic if/when I get that landed. * Bonus: sort compaction now supports nested sort keys. * I've added `BenchmarkSortRecordBatch` and `BenchmarkAppendSorted` to validate perf. * I benchmarked appending 1M rows against main: top-level keys run as fast as before, and a nested float64 key takes ~699 ms vs ~402 ms unsorted. Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
This diff adds support for sorting by nested fields.
I noticed that the contributing guidelines suggest opening a thread on the mailing list for "significant" features; I wasn't sure if this qualified as significant but I'm happy start a thread there if the maintainers feel it's appropriate.
Rationale for this change
The primary motivation for this change is enabling support for nested field sorting in iceberg-go (which is supported by the Python and Java implementations).
What changes are included in this PR?
compute.SortKeynow has aColumnPathfield that can be used to fully address nested struct fields.SortKey.ColumnIndexis preserved for backwards compat/ergo.ColumnPathtakes precedence overColumnIndexif both are specified.sort_indicesnow walks theColumnPathif it is specified.compute.SortRecordBatch.maskedStructFieldis basically a copy ofStruct.newStructFieldWithParentValidityMask. I opted to copy this rather than exporting a increasing the surface area of the public API.Are these changes tested?
ci/scripts/test.sh $(pwd)after fetching the submodules.Are there any user-facing changes?
compute.SortKeyhas a newColumnPathargument.