Skip to content

feat(arrow/compute): adds support for nested sort columns - #1337

Open
sgg wants to merge 2 commits into
apache:mainfrom
sgg:sgg/nested-field-sorting
Open

sgg wants to merge 2 commits into
apache:mainfrom
sgg:sgg/nested-field-sorting

Conversation

@sgg

@sgg sgg commented Sep 24, 2026 •

Copy link
Copy Markdown

This diff adds support for sorting by nested fields.

I noticed that the contributing guidelines suggest opening a thread on the mailing list for "significant" features; I wasn't sure if this qualified as significant but I'm happy start a thread there if the maintainers feel it's appropriate.

Rationale for this change

The primary motivation for this change is enabling support for nested field sorting in iceberg-go (which is supported by the Python and Java implementations).

What changes are included in this PR?

  • compute.SortKey now has a ColumnPath field that can be used to fully address nested struct fields. SortKey.ColumnIndex is preserved for backwards compat/ergo. ColumnPath takes precedence over ColumnIndex if both are specified.
  • sort_indices now walks the ColumnPath if it is specified.
  • I've added some new benchmarks for compute.SortRecordBatch.
  • Note maskedStructField is basically a copy of Struct.newStructFieldWithParentValidityMask. I opted to copy this rather than exporting a increasing the surface area of the public API.

Are these changes tested?

  • I've added some new unit tests and benchmarks.
  • I ran ci/scripts/test.sh $(pwd) after fetching the submodules.

Are there any user-facing changes?

  • compute.SortKey has a new ColumnPath argument.

This diff adds support for sorting by nested fields. The primary
motivation for this change is enabling support for sorting by nested
fields in iceberg-go, which is supported by the Python and Java
implementations.

* `compute.SortKey` now has a `ColumnPath` field that can be used to
  fully address nested struct fields. `SortKey.ColumnIndex` is
  preserved for backwards compat/ergo. `ColumnPath` takes precedence
  over `ColumnIndex` if both are specified.
* `sort_indices` now walks the `ColumnPath` if it is specified.
* I've added some new benchmarks for `compute.SortRecordBatch`.
* **Note** `maskedStructField` is basically a copy of
  `Struct.newStructFieldWithParentValidityMask`. I opted to copy this
  rather than exporting a increasing the surface area of the public API.

Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
@sgg
sgg requested a review from zeroshade as a code owner September 24, 2026 20:03

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: nested-path sorting returns wrong order on sliced input, and the new tests don't exercise parent-validity masking.

  1. Sliced input sorts incorrectly (inline on vector_sort.go:154).
  2. Masking is untested: the tests still pass with maskedStructField forced to return the raw child (inline on the test).
  3. SortKey no longer supports ==: it has been public since v18.6.0 and ColumnPath []int makes it non-comparable (inline on ColumnPath). This needs a deliberate API decision; a mailing-list thread seems worthwhile, as you offered. Arrow C++ SortKey uses a FieldRef target, which may be the more consistent shape.

Minor: the SortRecordBatch/SortTable docs still only mention ColumnIndex, and SortIndicesArray/SortIndicesChunked silently ignore ColumnPath.

FYI, not caused by this PR: kernels.SortIndices never releases its tmpBuf scratch allocations, so a CheckedAllocator AssertSize(t, 0) on any nullable sort key reports a 64-byte leak on main. I'll handle that separately.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/compute/vector_sort.go Outdated
Comment on lines +144 to +154
// Normalize to a zero offset so maskedNullBitmapBytes (indexed from
// fieldOffset above) lines up with the sliced buffers.
sliced := array.NewSliceData(field.Data(), 0, int64(field.Len()))
defer sliced.Release()

origBufs := sliced.Buffers()
bufs := make([]*memory.Buffer, len(origBufs))
copy(bufs, origBufs)
bufs[0] = memory.NewBufferBytes(maskedNullBitmapBytes)

data := array.NewData(sliced.DataType(), sliced.Len(), bufs, sliced.Children(), array.UnknownNullCount, 0)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NewSliceData(d, 0, n) keeps d.Offset() and shares the buffers; it does not reset the offset to zero. Passing 0 here makes the new array read values from index 0 and misaligns the bitmap, which was built at fieldOffset+i.

Repro: ids [100,101,1,2,3,4], ranks [-5,-4,30,null,10,20], then batch.NewSlice(2,6) and sort by {0,1,0}. The result is [1 2 3 4]; the expected order is [3 4 1 2]. Using sliced.Offset() fixes it. You could also skip the NewSliceData round-trip and build from field.Data() directly. Please add a sliced batch/chunk test.

Comment thread arrow/compute/vector_sort_test.go Outdated
Comment on lines +721 to +729

// Row 1: id=2, info=null. StructBuilder.Append(false) auto-appends
// null to info's children (rank), so rankBldr is not touched here;
// rank's underlying storage slot still ends up non-null-looking at
// the physical level, which is exactly what parent-validity masking
// must override.
bldr.Append(true)
idBldr.Append(2)
infoBldr.Append(false)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StructBuilder.Append(false) calls AppendNull() on every child (array/struct.go:395-398), so rank is already physically null here and the comment is inaccurate. As a result, this test and the multi-chunk table test pass even when maskedStructField just returns st.Field(i) unmasked (I checked).

Please build a case where the parent is null but the child slot is valid, e.g. with array.NewStructArrayWithNulls or NewStructData and a hand-built bitmap.

// the descendant's own physical validity bitmap. Only struct nesting is
// supported; a path element pointing into a list, map, or union column
// is an error.
ColumnPath []int

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a slice field makes SortKey non-comparable, so existing key == other / map-key usage stops compiling. SortKey has been public since v18.6.0, so this is a breaking change in a minor release. It needs a deliberate call. Arrow C++ SortKey takes a FieldRef target, which may be the more consistent shape.

Comment thread arrow/compute/vector_sort.go Outdated
leafType = resolved.DataType()
}
if leafType == nil {
leafType = chunked.DataType()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With zero chunks, the path is never validated and leafType falls back to the struct type. Validating the path once against the column type (compute.FieldPath.GetFieldFromType) before the per-chunk loop would give the correct leaf type and catch bad paths even on empty tables.

Comment thread arrow/compute/vector_sort.go Outdated
}
idx := path[0]
if idx < 0 || idx >= st.NumField() {
return nil, fmt.Errorf("%w: sort key struct field index %d out of range", arrow.ErrIndex, idx)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: an out-of-range top-level index returns ErrInvalid, but an out-of-range nested index returns ErrIndex. Pick one.

* adds additional tests to repro bugs found in review
* fixes offset bug in maskedStructField
* makes resolveSortColumnPath return ErrInvalid in both err cases
* evaluate leaf type once rather than per chunk
@sgg

sgg commented Sep 29, 2026

Copy link
Copy Markdown
Author

I appreciate the review @zeroshade . I didn't consider direct comparability of SortKey; I'll start a convo on the mailing list.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. 1ea3b385 fixes every code issue from the last round. I checked each one:

  • Sliced input: fixed. The new sliced batch and chunk tests pass.
  • Masking: fixed. Forcing maskedStructField to return the child unmasked now makes 4 of the nested tests fail, so they catch it.
  • Zero-chunk path validation and leaf type: fixed.
  • ErrIndex/ErrInvalid: fixed.

Still open:

  1. SortKey comparability (inline). Blocked on the mailing-list thread.
  2. SortIndicesArray/SortIndicesChunked ignore ColumnPath. With a struct array and ColumnPath: {0,1}, they return not implemented: sorting not supported for type STRUCT, which is misleading. Either resolve the path for these inputs or reject a non-empty ColumnPath with ErrInvalid and document it. (The lines that reset key.ColumnIndex = 0 aren't in the diff, so this can't go inline.)
  3. Docs: the SortRecordBatch/SortTable examples only show ColumnIndex. Please add a ColumnPath example.

// the descendant's own physical validity bitmap. Only struct nesting is
// supported; a path element pointing into a list, map, or union column
// is an error.
ColumnPath []int

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open: []int makes SortKey non-comparable. Holding this until the mailing-list thread settles the shape (slice vs. a C++-style FieldRef).

sgg added a commit to sgg/iceberg-go that referenced this pull request Nov 19, 2026
Today writing to a table whose default sort order sources a key from a
nested field fails, because the writer sorts each batch with
`compute.SortRecordBatch` and arrow-go's `compute.SortKey` can only
address top-level columns.

This diff adds support for nested sort keys by falling back to the row
comparator from sort compaction (apache#2345) when a key is nested in a
struct. The comparator resolves each key to a struct path and treats a
null ancestor struct as a null key. Sort orders whose keys are all
top-level columns still use arrow-go, so that we pick up any
improvements made there (e.g. the radix sort added in v18.9.0). The
fallback is intended to only stick around until arrow-go can provide
support for sorting by nested fields.

Notes:

* This implementation is entirely internal to the table package so we
  should be able to swap it out later.
* I am working on getting support for nested sort keys added to arrow-go
  (see apache/arrow-go#1337).
* Bonus: compaction operations now support nested sort keys.
* Sources under lists or maps are rejected w/ `ErrInvalidArgument`
  as they are not allowed by the Iceberg spec.
* We now set `FloatTotalOrder` on arrow's sort keys so top-level float
  keys match the Iceberg spec's float order, same as the comparator.
  Previously, arrow placed NaN next to nulls and treated -0 as equal to 0.
* Adds new benchmarks.
* I benchmarked appending 1M rows against main: top-level keys run as fast
  as before, and a nested float64 key takes ~699 ms vs ~402 ms unsorted.
  Total order also speeds up float keys: a 64K-row float64 batch w/o nulls
  sorts in ~2.0 ms instead of ~18.5 ms.

Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
sgg added a commit to sgg/iceberg-go that referenced this pull request Nov 19, 2026
This diff adds support for writing to tables that define sort keys from
nested struct fields.

Previously attempting to write to a table with such a sort order defined
would error. With this diff:

* Tables whose sort keys are all top-level fields use arrow-go directly,
  same as before.
* Tables with any sort key on a nested field fall back to the row
  comparator used by sort compaction (introduced with apache#2345).
* This implementation is entirely internal to the table package so we
  should be able to swap it out later.
* Sources under lists or maps are rejected w/ `ErrInvalidArgument`
  as they are not allowed by the Iceberg spec.
* We now set `FloatTotalOrder` on arrow's sort keys so top-level float
  keys match the Iceberg spec's float order, same as the comparator.
  Previously, arrow placed NaN next to nulls and treated -0 as equal to
  0, so tables w/ a top-level float sort key now write rows w/ NaN or -0
  keys in a different order. This is also faster: a 64K-row float64
  batch w/o nulls sorts in ~2.0 ms instead of ~18.5 ms.

Other Notes:

* Long term we should add support to arrow-go. I am working on that (see
  apache/arrow-go#1337) and plan to follow up w/ a change removing this
  logic if/when I get that landed.
* Bonus: sort compaction now supports nested sort keys.
* I've added `BenchmarkSortRecordBatch` and `BenchmarkAppendSorted` to
  validate perf.
* I benchmarked appending 1M rows against main: top-level keys run as
  fast as before, and a nested float64 key takes ~699 ms vs ~402 ms
  unsorted.

Signed-off-by: Samani G. Gikandi <samani@gojulas.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants