Skip to content

fix(arrow/array): avoid int overflow in Validate and ValidateFull on 32 bit platforms - #1619

Open
hcrosse wants to merge 2 commits into
apache:mainfrom
hcrosse:array-view-validate-32bit
Open

hcrosse wants to merge 2 commits into
apache:mainfrom
hcrosse:array-view-validate-32bit

Conversation

@hcrosse

@hcrosse hcrosse commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

On 32 bit platforms, validation accepts invalid arrays or panics when int overflows.

What changes are included in this PR?

These checks now compare in int64: view ranges and view buffer size, list view offset counts, Struct child lengths, run end coverage, and LargeBinary and LargeString first and last offsets. Error formats are unchanged.

The existing TestStructValidateRejectsOffsetLengthOverflow covers the Struct check. It fails on 32 bit platforms without this change and passes with it.

The 386 CI job now also runs the Validate tests in arrow/array, because they only catch these bugs on 32 bit platforms.

Are these changes tested?

Yes. I added tests for overflowing view ranges, view and list view layouts, run end coverage and large offsets.

Are there any user-facing changes?

Yes. Validation returns errors for these arrays on 32 bit platforms instead of accepting them or panicking. There is no API change.

Closes #1616.

…32 bit platforms

On platforms where int has 32 bits, such as GOARCH=386 and GOARCH=arm, validateViewValues added a view's buffer offset and size as int before comparing the sum with the buffer length. When the sum exceeded MaxInt32 it wrapped negative, passed the bounds check, and ValidateFull panicked with slice bounds out of range instead of returning an error. The same input returns an error on 64 bit platforms.

Compare in int64 instead, as Arrow C++ does in ValidateBinaryView. The error formats are unchanged. The same change applies to validateViewLayout, the child length check in Struct.Validate, the last run end check in validateRunEndEncoded and the offsets buffer size check for list view arrays. Each of them accepted an offset plus length beyond MaxInt32, and the view layout check also made ValidateFull panic.

LargeBinary and LargeString Validate truncated int64 offsets with int before comparing them with the data length. On 32 bit platforms an out of range offset was accepted for LargeBinary and made ValidateFull panic for LargeString. They now compare as int64, which matches C++ where large offsets are int64.

Most of the new rows fail without this change only on 32 bit platforms, and they pass on 64 bit platforms either way. Three rows only guard the rewritten comparisons and pass without this change on every platform. They are the range one byte past the referenced buffer and the two rows with large offsets equal to the data length.
@hcrosse
hcrosse marked this pull request as ready for review November 18, 2026 21:11
@hcrosse
hcrosse requested a review from zeroshade as a code owner November 18, 2026 21:11
Copilot AI balanced review requested due to automatic review settings November 18, 2026 21:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the array-validation overflow changes.

No open issues 🎉

What changed in this PR

This PR addresses #1616 by preventing integer overflow in Arrow array validation on 32-bit platforms, so affected invalid arrays return errors instead of passing validation or panicking.

Changes:

  • Use int64 arithmetic for view bounds, list-view offset counts, struct child lengths, and run-end coverage.
  • Preserve 64-bit offsets when validating LargeBinary and LargeString buffer bounds.
  • Add regression tests for overflow cases and boundary conditions without changing public APIs.
File Description
arrow/​array/​validate_test.go Adds overflow and boundary tests for binary, string, and view arrays.
arrow/​array/​struct.go Widens expected child-length arithmetic to int64.
arrow/​array/​string.go Avoids narrowing large string offsets during validation.
arrow/​array/​list.go Computes required offset counts in int64.
arrow/​array/​encoded.go Uses a widened end position for run-end coverage checks.
arrow/​array/​encoded_test.go Adds a regression test for overflowing run-end coverage.
arrow/​array/​binary.go Widens view bounds checks and preserves large binary offsets.

Review effort: Balanced

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the int64 checks cover every case in #1616. I cross-compiled the arrow/array tests for linux/386 and ran them under emulation. On the base commit, the new LargeBinary/LargeString, ListView/LargeListView and RunEndEncoded tests and the existing TestStructValidateRejectsOffsetLengthOverflow fail, and the view tests panic with slice bounds out of range [:-2147483632]. With this change they all pass. On 64-bit, go test -race ./arrow/array/ passes.

One change before this goes in, inline: these tests only catch the bug on 32 bit platforms, and CI never runs them there.

Comment thread arrow/array/validate_test.go Resolved
The overflow tests can only fail on 32 bit platforms, so no 64 bit job would catch a regression. The whole package can replace the Validate filter once the TestTimeJSON failure on 386 is fixed.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The new Test arrow/array validation for 386 step passes in the 386 job on this head, so CI now runs these overflow tests on 32 bit, where they catch the bug. The rest of the change is the same as what I checked under linux/386 emulation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow/array: ValidateFull panics on view ranges that overflow int on 32 bit platforms

3 participants