From 6f5a113b230738dee3853abeb42ccb49d9c188ce Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 28 Oct 2026 02:51:24 +0000 Subject: [PATCH 01/21] build(deps): bump the maven-minor-patch group with 14 updates Bumps the maven-minor-patch group with 14 updates: | Package | From | To | | --- | --- | --- | | [org.tukaani:xz](https://github.com/tukaani-project/xz-java) | `1.9` | `1.12` | | [org.hibernate.validator:hibernate-validator](https://github.com/hibernate/hibernate-validator) | `9.1.4.Final` | `9.1.5.Final` | | [it.unimi.dsi:fastutil](https://github.com/vigna/fastutil) | `8.5.4` | `8.5.19` | | [it.unimi.dsi:fastutil-core](https://github.com/vigna/fastutil) | `8.5.4` | `8.5.19` | | [com.github.ben-manes.caffeine:caffeine](https://github.com/ben-manes/caffeine) | `3.3.0` | `3.3.1` | | [net.bytebuddy:byte-buddy](https://github.com/raphw/byte-buddy) | `1.18.14` | `1.18.15` | | [net.bytebuddy:byte-buddy-agent](https://github.com/raphw/byte-buddy) | `1.18.14` | `1.18.15` | | [com.github.spotbugs:spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin) | `4.10.4.1` | `4.10.5.0` | | org.codehaus.mojo:javacc-maven-plugin | `2.4` | `2.6` | | org.apache.kafka:kafka-clients | `4.3.1` | `4.4.0` | | [com.tdunning:t-digest](https://github.com/tdunning/t-digest) | `3.2` | `3.3` | | [io.grpc:grpc-bom](https://github.com/grpc/grpc-java) | `1.84.2` | `1.84.3` | | [io.grpc:grpc-stub](https://github.com/grpc/grpc-java) | `1.84.2` | `1.84.3` | | [io.grpc:grpc-netty-shaded](https://github.com/grpc/grpc-java) | `1.84.2` | `1.84.3` | Updates `org.tukaani:xz` from 1.9 to 1.12 - [Release notes](https://github.com/tukaani-project/xz-java/releases) - [Changelog](https://github.com/tukaani-project/xz-java/blob/master/NEWS.md) - [Commits](https://github.com/tukaani-project/xz-java/compare/v1.9...v1.12) Updates `org.hibernate.validator:hibernate-validator` from 9.1.4.Final to 9.1.5.Final - [Release notes](https://github.com/hibernate/hibernate-validator/releases) - [Changelog](https://github.com/hibernate/hibernate-validator/blob/9.1.5.Final/changelog.md) - [Commits](https://github.com/hibernate/hibernate-validator/compare/9.1.4.Final...9.1.5.Final) Updates `it.unimi.dsi:fastutil` from 8.5.4 to 8.5.19 - [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES) - [Commits](https://github.com/vigna/fastutil/compare/8.5.4...8.5.19) Updates `it.unimi.dsi:fastutil-core` from 8.5.4 to 8.5.19 - [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES) - [Commits](https://github.com/vigna/fastutil/compare/8.5.4...8.5.19) Updates `it.unimi.dsi:fastutil-core` from 8.5.4 to 8.5.19 - [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES) - [Commits](https://github.com/vigna/fastutil/compare/8.5.4...8.5.19) Updates `com.github.ben-manes.caffeine:caffeine` from 3.3.0 to 3.3.1 - [Release notes](https://github.com/ben-manes/caffeine/releases) - [Commits](https://github.com/ben-manes/caffeine/compare/v3.3.0...v3.3.1) Updates `net.bytebuddy:byte-buddy` from 1.18.14 to 1.18.15 - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](https://github.com/raphw/byte-buddy/compare/byte-buddy-1.18.14...byte-buddy-1.18.15) Updates `net.bytebuddy:byte-buddy-agent` from 1.18.14 to 1.18.15 - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](https://github.com/raphw/byte-buddy/compare/byte-buddy-1.18.14...byte-buddy-1.18.15) Updates `net.bytebuddy:byte-buddy-agent` from 1.18.14 to 1.18.15 - [Release notes](https://github.com/raphw/byte-buddy/releases) - [Changelog](https://github.com/raphw/byte-buddy/blob/master/release-notes.md) - [Commits](https://github.com/raphw/byte-buddy/compare/byte-buddy-1.18.14...byte-buddy-1.18.15) Updates `com.github.spotbugs:spotbugs-maven-plugin` from 4.10.4.1 to 4.10.5.0 - [Release notes](https://github.com/spotbugs/spotbugs-maven-plugin/releases) - [Commits](https://github.com/spotbugs/spotbugs-maven-plugin/compare/spotbugs-maven-plugin-4.10.4.1...spotbugs-maven-plugin-4.10.5.0) Updates `org.codehaus.mojo:javacc-maven-plugin` from 2.4 to 2.6 Updates `org.apache.kafka:kafka-clients` from 4.3.1 to 4.4.0 Updates `com.tdunning:t-digest` from 3.2 to 3.3 - [Changelog](https://github.com/tdunning/t-digest/blob/main/RELEASE-NOTES.md) - [Commits](https://github.com/tdunning/t-digest/commits) Updates `io.grpc:grpc-bom` from 1.84.2 to 1.84.3 - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](https://github.com/grpc/grpc-java/compare/v1.84.2...v1.84.3) Updates `io.grpc:grpc-stub` from 1.84.2 to 1.84.3 - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](https://github.com/grpc/grpc-java/compare/v1.84.2...v1.84.3) Updates `io.grpc:grpc-netty-shaded` from 1.84.2 to 1.84.3 - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](https://github.com/grpc/grpc-java/compare/v1.84.2...v1.84.3) Updates `io.grpc:grpc-stub` from 1.84.2 to 1.84.3 - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](https://github.com/grpc/grpc-java/compare/v1.84.2...v1.84.3) Updates `io.grpc:grpc-netty-shaded` from 1.84.2 to 1.84.3 - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](https://github.com/grpc/grpc-java/compare/v1.84.2...v1.84.3) --- updated-dependencies: - dependency-name: org.tukaani:xz dependency-version: '1.12' dependency-type: direct:production update-type: version-update:semver-minor dependency-group: maven-minor-patch - dependency-name: org.hibernate.validator:hibernate-validator dependency-version: 9.1.5.Final dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: it.unimi.dsi:fastutil dependency-version: 8.5.19 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: it.unimi.dsi:fastutil-core dependency-version: 8.5.19 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: it.unimi.dsi:fastutil-core dependency-version: 8.5.19 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: com.github.ben-manes.caffeine:caffeine dependency-version: 3.3.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: net.bytebuddy:byte-buddy dependency-version: 1.18.15 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: net.bytebuddy:byte-buddy-agent dependency-version: 1.18.15 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: net.bytebuddy:byte-buddy-agent dependency-version: 1.18.15 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: com.github.spotbugs:spotbugs-maven-plugin dependency-version: 4.10.5.0 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: org.codehaus.mojo:javacc-maven-plugin dependency-version: '2.6' dependency-type: direct:production update-type: version-update:semver-minor dependency-group: maven-minor-patch - dependency-name: org.apache.kafka:kafka-clients dependency-version: 4.4.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: maven-minor-patch - dependency-name: com.tdunning:t-digest dependency-version: '3.3' dependency-type: direct:production update-type: version-update:semver-minor dependency-group: maven-minor-patch - dependency-name: io.grpc:grpc-bom dependency-version: 1.84.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: io.grpc:grpc-stub dependency-version: 1.84.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: io.grpc:grpc-netty-shaded dependency-version: 1.84.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: io.grpc:grpc-stub dependency-version: 1.84.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch - dependency-name: io.grpc:grpc-netty-shaded dependency-version: 1.84.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: maven-minor-patch ... Signed-off-by: dependabot[bot] --- .../druid-deltalake-extensions/pom.xml | 2 +- extensions-contrib/opentelemetry-emitter/pom.xml | 2 +- extensions-contrib/tdigestsketch/pom.xml | 2 +- pom.xml | 16 ++++++++-------- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/extensions-contrib/druid-deltalake-extensions/pom.xml b/extensions-contrib/druid-deltalake-extensions/pom.xml index 6d5dfc11c383..0aef0752d86e 100644 --- a/extensions-contrib/druid-deltalake-extensions/pom.xml +++ b/extensions-contrib/druid-deltalake-extensions/pom.xml @@ -116,7 +116,7 @@ it.unimi.dsi fastutil-core - 8.5.4 + 8.5.19 provided diff --git a/extensions-contrib/opentelemetry-emitter/pom.xml b/extensions-contrib/opentelemetry-emitter/pom.xml index 45335d65e4f8..5395aa29de32 100644 --- a/extensions-contrib/opentelemetry-emitter/pom.xml +++ b/extensions-contrib/opentelemetry-emitter/pom.xml @@ -39,7 +39,7 @@ 33.7.2-jre - 1.84.2 + 1.84.3 diff --git a/extensions-contrib/tdigestsketch/pom.xml b/extensions-contrib/tdigestsketch/pom.xml index e476fd2ffa48..1b5287361598 100644 --- a/extensions-contrib/tdigestsketch/pom.xml +++ b/extensions-contrib/tdigestsketch/pom.xml @@ -56,7 +56,7 @@ com.tdunning t-digest - 3.2 + 3.3 com.google.guava diff --git a/pom.xml b/pom.xml index b60ffd806d90..7e8cb298545f 100644 --- a/pom.xml +++ b/pom.xml @@ -75,7 +75,7 @@ UTF-8 2.0.24 5.9.0 - 4.3.1 + 4.4.0 2.9.0 @@ -85,7 +85,7 @@ 1.29.0 1.12.2 1.18.1 - 1.18.14 + 1.18.15 @@ -99,7 +99,7 @@ 2.50.0 2.50.0 8.0.0 - 8.5.4 + 8.5.19 33.7.2-jre 6.0.0 3.0 @@ -128,10 +128,10 @@ 1.12.797 2.55.6 2.3.9 - 3.3.0 + 3.3.1 0.8.15 2.0.5 - 9.1.4.Final + 9.1.5.Final 4.5.14 5.5.0 25.0.0-legacy @@ -726,7 +726,7 @@ org.tukaani xz - 1.9 + 1.12 com.github.luben @@ -1715,7 +1715,7 @@ com.github.spotbugs spotbugs-maven-plugin - 4.10.4.1 + 4.10.5.0 codestyle/spotbugs-exclude.xml 768 @@ -2126,7 +2126,7 @@ org.codehaus.mojo javacc-maven-plugin - 2.4 + 2.6 org.codehaus.mojo From 9b1da70c81093a51ce1e5e4154e67d79a43a8f7d Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 11:22:30 +0800 Subject: [PATCH 02/21] fix: avoid fastutil grow override conflict --- .../druid/collections/fastutil/DruidIntList.java | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java index d732c9723567..cf8560380a32 100644 --- a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java +++ b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java @@ -32,14 +32,14 @@ public DruidIntList(int capacity) public void addArray(int[] vals) { - grow(size + vals.length); + growDruid(size + vals.length); System.arraycopy(vals, 0, a, size, vals.length); size += vals.length; } public void fill(int val, int count) { - grow(size + count); + growDruid(size + count); java.util.Arrays.fill(a, size, size + count, val); size += count; } @@ -47,7 +47,7 @@ public void fill(int val, int count) public void fillWithRepeat(int[] vals, int repeat) { int count = vals.length * repeat; - grow(size + count); + growDruid(size + count); for (int i = 0; i < count; i += vals.length) { System.arraycopy(vals, 0, a, size + i, vals.length); } @@ -61,7 +61,7 @@ public void fillRuns(int[] vals, int runLength, int repeat) return; } int count = vals.length * runLength * repeat; - grow(size + count); + growDruid(size + count); for (int i = 0; i < repeat; ++i) { for (int val : vals) { // there's a += hidden in there, don't be tricked! @@ -83,7 +83,7 @@ public void resetToSize(int targetSize) * @see IntArrayList#grow(int) */ @SuppressWarnings("ArrayEquality") - private void grow(int capacity) + private void growDruid(int capacity) { if (capacity <= a.length) { return; From e0a48a50c9d1f2a5b0e5b6df9a5d8f87e0140123 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 11:33:28 +0800 Subject: [PATCH 03/21] build: align zstd with kafka client --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7e8cb298545f..3470e6a7edb7 100644 --- a/pom.xml +++ b/pom.xml @@ -731,7 +731,7 @@ com.github.luben zstd-jni - 1.5.7-11 + 1.5.7-17 com.fasterxml.jackson From f8b681b0784aa190b2365a9df12ffc156758e5df Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 11:58:39 +0800 Subject: [PATCH 04/21] build: update dependency license registry --- licenses.yaml | 14 +++++++------- processing/pom.xml | 4 ---- 2 files changed, 7 insertions(+), 11 deletions(-) diff --git a/licenses.yaml b/licenses.yaml index 1221c6f98f8c..c4963f48d529 100644 --- a/licenses.yaml +++ b/licenses.yaml @@ -446,7 +446,7 @@ name: Caffeine license_category: binary module: java-core license_name: Apache License version 2.0 -version: 3.3.0 +version: 3.3.1 libraries: - com.github.ben-manes.caffeine: caffeine @@ -1745,7 +1745,7 @@ name: fastutil license_category: binary module: java-core license_name: Apache License version 2.0 -version: 8.5.4 +version: 8.5.19 libraries: - it.unimi.dsi: fastutil - it.unimi.dsi: fastutil-core @@ -2577,7 +2577,7 @@ name: Hibernate Validator license_category: binary module: java-core license_name: Apache License version 2.0 -version: 9.1.4.Final +version: 9.1.5.Final libraries: - org.hibernate.validator: hibernate-validator @@ -3497,7 +3497,7 @@ notices: --- name: Apache Kafka -version: 4.3.1 +version: 4.4.0 license_category: binary module: extensions/druid-kafka-indexing-service license_name: Apache License version 2.0 @@ -3685,7 +3685,7 @@ name: JNI binding for Zstd license_category: binary module: java-core license_name: BSD-2-Clause License -version: 1.5.7-11 +version: 1.5.7-17 copyright: Luben Karavelov license_file_path: licenses/bin/zstd-jni.BSD2 libraries: @@ -4051,7 +4051,7 @@ name: XZ license_category: binary module: java-core license_name: Public Domain -version: 1.9 +version: 1.12 libraries: - org.tukaani: xz @@ -4702,7 +4702,7 @@ name: Apache Kafka license_category: binary module: extensions/druid-kafka-extraction-namespace license_name: Apache License version 2.0 -version: 4.3.1 +version: 4.4.0 libraries: - org.apache.kafka: kafka-clients notices: diff --git a/processing/pom.xml b/processing/pom.xml index b3a04ce8f23a..79c1fc0dfd80 100644 --- a/processing/pom.xml +++ b/processing/pom.xml @@ -233,10 +233,6 @@ it.unimi.dsi fastutil - - it.unimi.dsi - fastutil-core - org.antlr antlr4-runtime From cb12ea90c55d75a4267d6a0ab3e5b486aab351ef Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 12:20:19 +0800 Subject: [PATCH 05/21] build: declare fastutil in server --- server/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server/pom.xml b/server/pom.xml index 42282f03832b..0e9f1cb79990 100644 --- a/server/pom.xml +++ b/server/pom.xml @@ -175,7 +175,7 @@ it.unimi.dsi - fastutil-core + fastutil com.github.ben-manes.caffeine From 7cd23fafd6c94af97bcbd840283577535378b50f Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 12:38:43 +0800 Subject: [PATCH 06/21] build: align indexing service fastutil dependency --- indexing-service/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/indexing-service/pom.xml b/indexing-service/pom.xml index 150d3a019a11..218da4caa204 100644 --- a/indexing-service/pom.xml +++ b/indexing-service/pom.xml @@ -138,7 +138,7 @@ it.unimi.dsi - fastutil-core + fastutil org.apache.logging.log4j From 5456513fce0913271c1a00452a59b09ba148de02 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 12:54:53 +0800 Subject: [PATCH 07/21] build: align sql fastutil dependency --- sql/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql/pom.xml b/sql/pom.xml index 7e342d1e431a..56707abe63cb 100644 --- a/sql/pom.xml +++ b/sql/pom.xml @@ -161,7 +161,7 @@ it.unimi.dsi - fastutil-core + fastutil com.fasterxml.jackson.core From df99ecefdb3e082172a27f1e7df53ce47b33e896 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 13:12:19 +0800 Subject: [PATCH 08/21] build: align datasketches fastutil dependency --- extensions-core/datasketches/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-core/datasketches/pom.xml b/extensions-core/datasketches/pom.xml index 746b2d21406c..d086d891753e 100644 --- a/extensions-core/datasketches/pom.xml +++ b/extensions-core/datasketches/pom.xml @@ -99,7 +99,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From b0d5771e55fe5c3a4a129a37818664fb37644223 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 13:29:40 +0800 Subject: [PATCH 09/21] build: align histogram fastutil dependency --- extensions-core/histogram/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-core/histogram/pom.xml b/extensions-core/histogram/pom.xml index 9e119f562233..6d73479cca15 100644 --- a/extensions-core/histogram/pom.xml +++ b/extensions-core/histogram/pom.xml @@ -83,7 +83,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From cf12f968c3bdd688877fe2106d06110b7e253f2f Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 13:46:59 +0800 Subject: [PATCH 10/21] build: align benchmark fastutil dependency --- benchmarks/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/pom.xml b/benchmarks/pom.xml index f96bb0eca953..82d47ea4b0e7 100644 --- a/benchmarks/pom.xml +++ b/benchmarks/pom.xml @@ -170,7 +170,7 @@ it.unimi.dsi - fastutil-core + fastutil org.roaringbitmap From f46e2f52273a4e403cd6ce428b0eb8126c0c149e Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 14:11:00 +0800 Subject: [PATCH 11/21] build: align lookups fastutil dependency --- extensions-core/lookups-cached-global/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-core/lookups-cached-global/pom.xml b/extensions-core/lookups-cached-global/pom.xml index 0d584c22b9d3..81d2fac38dca 100644 --- a/extensions-core/lookups-cached-global/pom.xml +++ b/extensions-core/lookups-cached-global/pom.xml @@ -51,7 +51,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From da51c0915725de64f58470662d17af241db9357d Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 14:27:25 +0800 Subject: [PATCH 12/21] build: align S3 fastutil dependency --- extensions-core/s3-extensions/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-core/s3-extensions/pom.xml b/extensions-core/s3-extensions/pom.xml index 976ce809ea32..d13a746306cc 100644 --- a/extensions-core/s3-extensions/pom.xml +++ b/extensions-core/s3-extensions/pom.xml @@ -179,7 +179,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From b522ec28c671943c70484171ff7d4898b31aeaaa Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 14:57:28 +0800 Subject: [PATCH 13/21] fix: handle null dictionary values with fastutil 8.5.19 --- .../druid-exact-count-bitmap/pom.xml | 2 +- .../ComparatorSortedDimensionDictionary.java | 29 +++++++++++++++---- .../segment/SortedDimensionDictionary.java | 21 +++++++++++--- 3 files changed, 42 insertions(+), 10 deletions(-) diff --git a/extensions-contrib/druid-exact-count-bitmap/pom.xml b/extensions-contrib/druid-exact-count-bitmap/pom.xml index b45c93a25488..9b878d6498c4 100644 --- a/extensions-contrib/druid-exact-count-bitmap/pom.xml +++ b/extensions-contrib/druid-exact-count-bitmap/pom.xml @@ -74,7 +74,7 @@ it.unimi.dsi - fastutil-core + fastutil provided diff --git a/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java b/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java index 0bb1e28577ad..5d6a0b24fe39 100644 --- a/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java +++ b/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java @@ -25,6 +25,7 @@ import it.unimi.dsi.fastutil.objects.Object2IntRBTreeMap; import it.unimi.dsi.fastutil.objects.Object2IntSortedMap; +import java.util.ArrayList; import java.util.Comparator; import java.util.List; @@ -43,17 +44,35 @@ public class ComparatorSortedDimensionDictionary public ComparatorSortedDimensionDictionary(List idToValue, Comparator comparator, int length) { - Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(comparator); + final Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(comparator); + int nullId = -1; for (int id = 0; id < length; id++) { - T value = idToValue.get(id); - sortedMap.put(value, id); + final T value = idToValue.get(id); + if (value == null) { + nullId = id; + } else { + sortedMap.put(value, id); + } } this.sortedVals = Lists.newArrayList(sortedMap.keySet()); + int nullIndex = 0; + if (nullId >= 0) { + while (nullIndex < sortedVals.size() && comparator.compare(sortedVals.get(nullIndex), null) < 0) { + nullIndex++; + } + this.sortedVals.add(nullIndex, null); + } this.idToIndex = new int[length]; this.indexToId = new int[length]; - int index = 0; + final List sortedIds = new ArrayList<>(sortedMap.size() + (nullId >= 0 ? 1 : 0)); for (IntIterator iterator = sortedMap.values().iterator(); iterator.hasNext(); ) { - int id = iterator.nextInt(); + sortedIds.add(iterator.nextInt()); + } + if (nullId >= 0) { + sortedIds.add(nullIndex, nullId); + } + int index = 0; + for (final int id : sortedIds) { idToIndex[id] = index; indexToId[index] = id; index++; diff --git a/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java b/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java index fdd79b341a19..2f3f57e19f7e 100644 --- a/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java +++ b/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java @@ -39,17 +39,30 @@ public class SortedDimensionDictionary> public SortedDimensionDictionary(List idToValue, int length) { - Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(Comparators.naturalNullsFirst()); + final Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(Comparators.naturalNullsFirst()); + int nullId = -1; for (int id = 0; id < length; id++) { - T value = idToValue.get(id); - sortedMap.put(value, id); + final T value = idToValue.get(id); + if (value == null) { + nullId = id; + } else { + sortedMap.put(value, id); + } } this.sortedVals = Lists.newArrayList(sortedMap.keySet()); + if (nullId >= 0) { + this.sortedVals.add(0, null); + } this.idToIndex = new int[length]; this.indexToId = new int[length]; int index = 0; + if (nullId >= 0) { + idToIndex[nullId] = index; + indexToId[index] = nullId; + index++; + } for (IntIterator iterator = sortedMap.values().iterator(); iterator.hasNext(); ) { - int id = iterator.nextInt(); + final int id = iterator.nextInt(); idToIndex[id] = index; indexToId[index] = id; index++; From ffa25bb6de0d0568ecd149187c7d264d74dc2861 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 15:21:00 +0800 Subject: [PATCH 14/21] fix: declare fastutil dependency for tdigestsketch --- extensions-contrib/tdigestsketch/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-contrib/tdigestsketch/pom.xml b/extensions-contrib/tdigestsketch/pom.xml index 1b5287361598..e395bc245434 100644 --- a/extensions-contrib/tdigestsketch/pom.xml +++ b/extensions-contrib/tdigestsketch/pom.xml @@ -111,7 +111,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From 76073e8cd506dfd9722a75b411482d0926361d2d Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 15:41:31 +0800 Subject: [PATCH 15/21] fix: declare fastutil dependency for ddsketch --- extensions-contrib/ddsketch/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions-contrib/ddsketch/pom.xml b/extensions-contrib/ddsketch/pom.xml index bde1593cf99b..57c40af876e7 100644 --- a/extensions-contrib/ddsketch/pom.xml +++ b/extensions-contrib/ddsketch/pom.xml @@ -110,7 +110,7 @@ it.unimi.dsi - fastutil-core + fastutil provided From 512318cce60b7d75a510abfb55fac3a51a4251a5 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 15:54:48 +0800 Subject: [PATCH 16/21] build: replace fastutil-core with fastutil in remaining modules fastutil 8.5.19 ships all classes in the fastutil jar and no longer depends on fastutil-core, so modules declaring fastutil-core fail dependency:analyze. Also drop the copy of IntArrayList#grow from DruidIntList now that fastutil exposes it publicly. --- embedded-tests/pom.xml | 2 +- extensions-contrib/distinctcount/pom.xml | 5 --- .../druid-deltalake-extensions/pom.xml | 3 +- .../spectator-histogram/pom.xml | 5 --- extensions-core/orc-extensions/pom.xml | 5 --- multi-stage-query/pom.xml | 5 --- .../collections/fastutil/DruidIntList.java | 31 +++---------------- 7 files changed, 6 insertions(+), 50 deletions(-) diff --git a/embedded-tests/pom.xml b/embedded-tests/pom.xml index 620b187e3ab3..a8663aa06443 100644 --- a/embedded-tests/pom.xml +++ b/embedded-tests/pom.xml @@ -525,7 +525,7 @@ it.unimi.dsi - fastutil-core + fastutil test diff --git a/extensions-contrib/distinctcount/pom.xml b/extensions-contrib/distinctcount/pom.xml index ade118ab4f6f..dfabaa39bf82 100644 --- a/extensions-contrib/distinctcount/pom.xml +++ b/extensions-contrib/distinctcount/pom.xml @@ -90,11 +90,6 @@ fastutil provided - - it.unimi.dsi - fastutil-core - provided - diff --git a/extensions-contrib/druid-deltalake-extensions/pom.xml b/extensions-contrib/druid-deltalake-extensions/pom.xml index 0aef0752d86e..c7503bbfa100 100644 --- a/extensions-contrib/druid-deltalake-extensions/pom.xml +++ b/extensions-contrib/druid-deltalake-extensions/pom.xml @@ -115,8 +115,7 @@ it.unimi.dsi - fastutil-core - 8.5.19 + fastutil provided diff --git a/extensions-contrib/spectator-histogram/pom.xml b/extensions-contrib/spectator-histogram/pom.xml index 481f9d396360..3d98daf72733 100644 --- a/extensions-contrib/spectator-histogram/pom.xml +++ b/extensions-contrib/spectator-histogram/pom.xml @@ -101,11 +101,6 @@ jackson-annotations provided - - it.unimi.dsi - fastutil-core - provided - com.google.errorprone error_prone_annotations diff --git a/extensions-core/orc-extensions/pom.xml b/extensions-core/orc-extensions/pom.xml index ed60bdc9bd63..f1e3e78d7bef 100644 --- a/extensions-core/orc-extensions/pom.xml +++ b/extensions-core/orc-extensions/pom.xml @@ -177,11 +177,6 @@ fastutil provided - - it.unimi.dsi - fastutil-core - provided - com.google.guava guava diff --git a/multi-stage-query/pom.xml b/multi-stage-query/pom.xml index 97df5ed6ab40..57752fa24008 100644 --- a/multi-stage-query/pom.xml +++ b/multi-stage-query/pom.xml @@ -194,11 +194,6 @@ fastutil provided - - it.unimi.dsi - fastutil-core - provided - commons-io commons-io diff --git a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java index cf8560380a32..67fe2b517914 100644 --- a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java +++ b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java @@ -19,9 +19,7 @@ package org.apache.druid.collections.fastutil; -import it.unimi.dsi.fastutil.Arrays; import it.unimi.dsi.fastutil.ints.IntArrayList; -import it.unimi.dsi.fastutil.ints.IntArrays; public class DruidIntList extends IntArrayList { @@ -32,14 +30,14 @@ public DruidIntList(int capacity) public void addArray(int[] vals) { - growDruid(size + vals.length); + grow(size + vals.length); System.arraycopy(vals, 0, a, size, vals.length); size += vals.length; } public void fill(int val, int count) { - growDruid(size + count); + grow(size + count); java.util.Arrays.fill(a, size, size + count, val); size += count; } @@ -47,7 +45,7 @@ public void fill(int val, int count) public void fillWithRepeat(int[] vals, int repeat) { int count = vals.length * repeat; - growDruid(size + count); + grow(size + count); for (int i = 0; i < count; i += vals.length) { System.arraycopy(vals, 0, a, size + i, vals.length); } @@ -61,7 +59,7 @@ public void fillRuns(int[] vals, int runLength, int repeat) return; } int count = vals.length * runLength * repeat; - growDruid(size + count); + grow(size + count); for (int i = 0; i < repeat; ++i) { for (int val : vals) { // there's a += hidden in there, don't be tricked! @@ -75,25 +73,4 @@ public void resetToSize(int targetSize) a = new int[targetSize]; size = 0; } - - /** - * Method gratuitously "borrowed" from IntArrayList, would've been nice if that method wasn't private, but ah well. - * - * @param capacity the capacity to grow to - * @see IntArrayList#grow(int) - */ - @SuppressWarnings("ArrayEquality") - private void growDruid(int capacity) - { - if (capacity <= a.length) { - return; - } - if (a != IntArrays.DEFAULT_EMPTY_ARRAY) { - capacity = (int) Math.max(Math.min((long) a.length + (a.length >> 1), Arrays.MAX_ARRAY_SIZE), capacity); - } else if (capacity < DEFAULT_INITIAL_CAPACITY) { - capacity = DEFAULT_INITIAL_CAPACITY; - } - a = IntArrays.forceCapacity(a, capacity, size); - assert size <= a.length; - } } From 9c5fb440f9af70301b0cb577e63ef53cf7c8dec8 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 16:39:17 +0800 Subject: [PATCH 17/21] build: exclude kafka-clients, javacc-maven-plugin and spotbugs-maven-plugin upgrades - kafka-clients 4.4.0 requires zstd-jni 1.5.7-17, which cannot decode legacy zstd frames; keep kafka-clients 4.3.1 and zstd-jni 1.5.7-11. - javacc-maven-plugin 2.6 (JavaCC 5) changes SQL parse error messages. - spotbugs-maven-plugin 4.10.5.0 reports false UUF_UNUSED_FIELD bugs. --- licenses.yaml | 6 +++--- pom.xml | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/licenses.yaml b/licenses.yaml index c4963f48d529..c7509e4d84ad 100644 --- a/licenses.yaml +++ b/licenses.yaml @@ -3497,7 +3497,7 @@ notices: --- name: Apache Kafka -version: 4.4.0 +version: 4.3.1 license_category: binary module: extensions/druid-kafka-indexing-service license_name: Apache License version 2.0 @@ -3685,7 +3685,7 @@ name: JNI binding for Zstd license_category: binary module: java-core license_name: BSD-2-Clause License -version: 1.5.7-17 +version: 1.5.7-11 copyright: Luben Karavelov license_file_path: licenses/bin/zstd-jni.BSD2 libraries: @@ -4702,7 +4702,7 @@ name: Apache Kafka license_category: binary module: extensions/druid-kafka-extraction-namespace license_name: Apache License version 2.0 -version: 4.4.0 +version: 4.3.1 libraries: - org.apache.kafka: kafka-clients notices: diff --git a/pom.xml b/pom.xml index 3470e6a7edb7..6573630bae5a 100644 --- a/pom.xml +++ b/pom.xml @@ -75,7 +75,7 @@ UTF-8 2.0.24 5.9.0 - 4.4.0 + 4.3.1 2.9.0 @@ -731,7 +731,7 @@ com.github.luben zstd-jni - 1.5.7-17 + 1.5.7-11 com.fasterxml.jackson @@ -1715,7 +1715,7 @@ com.github.spotbugs spotbugs-maven-plugin - 4.10.5.0 + 4.10.4.1 codestyle/spotbugs-exclude.xml 768 @@ -2126,7 +2126,7 @@ org.codehaus.mojo javacc-maven-plugin - 2.6 + 2.4 org.codehaus.mojo From 57b7e2cc0c6793ad325f390596e6cf4fdbcd7c8a Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 16:48:18 +0800 Subject: [PATCH 18/21] test: update t-digest 3.3 expectations --- .../sql/TDigestSketchSqlAggregatorTest.java | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java b/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java index c18d0ad7f434..0b87eb560669 100644 --- a/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java +++ b/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/sql/TDigestSketchSqlAggregatorTest.java @@ -182,7 +182,7 @@ public void testCastedQuantileAndCompressionParamForTDigestQuantileAgg() ), ResultMatchMode.EQUALS_EPS, ImmutableList.of( - new Object[]{1.0, 3.5, 6.0} + new Object[]{1.0, 4.0, 6.0} ) ); } @@ -287,9 +287,9 @@ public void testComputingQuantileOnPreAggregatedSketch() final List expectedResults = ImmutableList.of( new Object[]{ - 1.1, - 2.9, - 5.3, + 1.0, + 3.0, + 5.0, 6.0 } ); @@ -331,7 +331,7 @@ public void testGeneratingSketchAndComputingQuantileOnFly() final List expectedResults = ImmutableList.of( new Object[]{ 1.0, - 3.5, + 4.0, 6.0 } ); @@ -387,7 +387,7 @@ public void testQuantileOnNumericValues() final List expectedResults = ImmutableList.of( new Object[]{ 1.0, - 3.5, + 4.0, 6.0 } ); @@ -451,7 +451,7 @@ public void testCompressionParamForTDigestQuantileAgg() ), ResultMatchMode.EQUALS_EPS, ImmutableList.of( - new Object[]{1.0, 3.5, 6.0} + new Object[]{1.0, 4.0, 6.0} ) ); } From 38abce5729793a678d224ef0ad25f289f6628f36 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 17:28:26 +0800 Subject: [PATCH 19/21] fix: support null array overlap constants --- .../src/main/java/org/apache/druid/math/expr/Function.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/processing/src/main/java/org/apache/druid/math/expr/Function.java b/processing/src/main/java/org/apache/druid/math/expr/Function.java index d2d9995d9c95..98e12596bac0 100644 --- a/processing/src/main/java/org/apache/druid/math/expr/Function.java +++ b/processing/src/main/java/org/apache/druid/math/expr/Function.java @@ -21,7 +21,6 @@ import com.google.common.base.Preconditions; import com.google.common.collect.ImmutableSet; -import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet; import it.unimi.dsi.fastutil.objects.ObjectArrays; import it.unimi.dsi.fastutil.objects.ObjectOpenHashSet; import org.apache.druid.annotations.SuppressFBWarnings; @@ -4254,7 +4253,7 @@ public ExprEval apply(List args, Expr.ObjectBinding bindings) } }; } - final Set set = new ObjectAVLTreeSet<>(lhsArrayType.getElementType().getNullableStrategy()); + final Set set = new TreeSet<>(lhsArrayType.getElementType().getNullableStrategy()); set.addAll(Arrays.asList(rhsArray)); return new OverlapConstantArray(set); } From 463eb7aec44b6ab6d0d7b2d47fee5d986579d7e8 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Wed, 28 Oct 2026 21:21:25 +0800 Subject: [PATCH 20/21] build: move the fastutil upgrade to a separate PR Keeps fastutil and fastutil-core at 8.5.4 here and restores the code and module dependency changes that the 8.5.19 upgrade required. The upgrade is handled on its own branch. Co-Authored-By: Claude Opus 5.5 --- benchmarks/pom.xml | 2 +- embedded-tests/pom.xml | 2 +- extensions-contrib/ddsketch/pom.xml | 2 +- extensions-contrib/distinctcount/pom.xml | 5 ++++ .../druid-deltalake-extensions/pom.xml | 3 +- .../druid-exact-count-bitmap/pom.xml | 2 +- .../spectator-histogram/pom.xml | 5 ++++ extensions-contrib/tdigestsketch/pom.xml | 2 +- extensions-core/datasketches/pom.xml | 2 +- extensions-core/histogram/pom.xml | 2 +- extensions-core/lookups-cached-global/pom.xml | 2 +- extensions-core/orc-extensions/pom.xml | 5 ++++ extensions-core/s3-extensions/pom.xml | 2 +- indexing-service/pom.xml | 2 +- licenses.yaml | 2 +- multi-stage-query/pom.xml | 5 ++++ pom.xml | 2 +- processing/pom.xml | 4 +++ .../collections/fastutil/DruidIntList.java | 23 +++++++++++++++ .../org/apache/druid/math/expr/Function.java | 3 +- .../ComparatorSortedDimensionDictionary.java | 29 ++++--------------- .../segment/SortedDimensionDictionary.java | 21 +++----------- server/pom.xml | 2 +- sql/pom.xml | 2 +- 24 files changed, 74 insertions(+), 57 deletions(-) diff --git a/benchmarks/pom.xml b/benchmarks/pom.xml index 82d47ea4b0e7..f96bb0eca953 100644 --- a/benchmarks/pom.xml +++ b/benchmarks/pom.xml @@ -170,7 +170,7 @@ it.unimi.dsi - fastutil + fastutil-core org.roaringbitmap diff --git a/embedded-tests/pom.xml b/embedded-tests/pom.xml index a8663aa06443..620b187e3ab3 100644 --- a/embedded-tests/pom.xml +++ b/embedded-tests/pom.xml @@ -525,7 +525,7 @@ it.unimi.dsi - fastutil + fastutil-core test diff --git a/extensions-contrib/ddsketch/pom.xml b/extensions-contrib/ddsketch/pom.xml index 57c40af876e7..bde1593cf99b 100644 --- a/extensions-contrib/ddsketch/pom.xml +++ b/extensions-contrib/ddsketch/pom.xml @@ -110,7 +110,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-contrib/distinctcount/pom.xml b/extensions-contrib/distinctcount/pom.xml index dfabaa39bf82..ade118ab4f6f 100644 --- a/extensions-contrib/distinctcount/pom.xml +++ b/extensions-contrib/distinctcount/pom.xml @@ -90,6 +90,11 @@ fastutil provided + + it.unimi.dsi + fastutil-core + provided + diff --git a/extensions-contrib/druid-deltalake-extensions/pom.xml b/extensions-contrib/druid-deltalake-extensions/pom.xml index c7503bbfa100..6d5dfc11c383 100644 --- a/extensions-contrib/druid-deltalake-extensions/pom.xml +++ b/extensions-contrib/druid-deltalake-extensions/pom.xml @@ -115,7 +115,8 @@ it.unimi.dsi - fastutil + fastutil-core + 8.5.4 provided diff --git a/extensions-contrib/druid-exact-count-bitmap/pom.xml b/extensions-contrib/druid-exact-count-bitmap/pom.xml index 9b878d6498c4..b45c93a25488 100644 --- a/extensions-contrib/druid-exact-count-bitmap/pom.xml +++ b/extensions-contrib/druid-exact-count-bitmap/pom.xml @@ -74,7 +74,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-contrib/spectator-histogram/pom.xml b/extensions-contrib/spectator-histogram/pom.xml index 3d98daf72733..481f9d396360 100644 --- a/extensions-contrib/spectator-histogram/pom.xml +++ b/extensions-contrib/spectator-histogram/pom.xml @@ -101,6 +101,11 @@ jackson-annotations provided + + it.unimi.dsi + fastutil-core + provided + com.google.errorprone error_prone_annotations diff --git a/extensions-contrib/tdigestsketch/pom.xml b/extensions-contrib/tdigestsketch/pom.xml index e395bc245434..1b5287361598 100644 --- a/extensions-contrib/tdigestsketch/pom.xml +++ b/extensions-contrib/tdigestsketch/pom.xml @@ -111,7 +111,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-core/datasketches/pom.xml b/extensions-core/datasketches/pom.xml index d086d891753e..746b2d21406c 100644 --- a/extensions-core/datasketches/pom.xml +++ b/extensions-core/datasketches/pom.xml @@ -99,7 +99,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-core/histogram/pom.xml b/extensions-core/histogram/pom.xml index 6d73479cca15..9e119f562233 100644 --- a/extensions-core/histogram/pom.xml +++ b/extensions-core/histogram/pom.xml @@ -83,7 +83,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-core/lookups-cached-global/pom.xml b/extensions-core/lookups-cached-global/pom.xml index 81d2fac38dca..0d584c22b9d3 100644 --- a/extensions-core/lookups-cached-global/pom.xml +++ b/extensions-core/lookups-cached-global/pom.xml @@ -51,7 +51,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/extensions-core/orc-extensions/pom.xml b/extensions-core/orc-extensions/pom.xml index f1e3e78d7bef..ed60bdc9bd63 100644 --- a/extensions-core/orc-extensions/pom.xml +++ b/extensions-core/orc-extensions/pom.xml @@ -177,6 +177,11 @@ fastutil provided + + it.unimi.dsi + fastutil-core + provided + com.google.guava guava diff --git a/extensions-core/s3-extensions/pom.xml b/extensions-core/s3-extensions/pom.xml index d13a746306cc..976ce809ea32 100644 --- a/extensions-core/s3-extensions/pom.xml +++ b/extensions-core/s3-extensions/pom.xml @@ -179,7 +179,7 @@ it.unimi.dsi - fastutil + fastutil-core provided diff --git a/indexing-service/pom.xml b/indexing-service/pom.xml index 218da4caa204..150d3a019a11 100644 --- a/indexing-service/pom.xml +++ b/indexing-service/pom.xml @@ -138,7 +138,7 @@ it.unimi.dsi - fastutil + fastutil-core org.apache.logging.log4j diff --git a/licenses.yaml b/licenses.yaml index c7509e4d84ad..54adf1835d3f 100644 --- a/licenses.yaml +++ b/licenses.yaml @@ -1745,7 +1745,7 @@ name: fastutil license_category: binary module: java-core license_name: Apache License version 2.0 -version: 8.5.19 +version: 8.5.4 libraries: - it.unimi.dsi: fastutil - it.unimi.dsi: fastutil-core diff --git a/multi-stage-query/pom.xml b/multi-stage-query/pom.xml index 57752fa24008..97df5ed6ab40 100644 --- a/multi-stage-query/pom.xml +++ b/multi-stage-query/pom.xml @@ -194,6 +194,11 @@ fastutil provided + + it.unimi.dsi + fastutil-core + provided + commons-io commons-io diff --git a/pom.xml b/pom.xml index 6573630bae5a..10e916a73982 100644 --- a/pom.xml +++ b/pom.xml @@ -99,7 +99,7 @@ 2.50.0 2.50.0 8.0.0 - 8.5.19 + 8.5.4 33.7.2-jre 6.0.0 3.0 diff --git a/processing/pom.xml b/processing/pom.xml index 79c1fc0dfd80..b3a04ce8f23a 100644 --- a/processing/pom.xml +++ b/processing/pom.xml @@ -233,6 +233,10 @@ it.unimi.dsi fastutil + + it.unimi.dsi + fastutil-core + org.antlr antlr4-runtime diff --git a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java index 67fe2b517914..d732c9723567 100644 --- a/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java +++ b/processing/src/main/java/org/apache/druid/collections/fastutil/DruidIntList.java @@ -19,7 +19,9 @@ package org.apache.druid.collections.fastutil; +import it.unimi.dsi.fastutil.Arrays; import it.unimi.dsi.fastutil.ints.IntArrayList; +import it.unimi.dsi.fastutil.ints.IntArrays; public class DruidIntList extends IntArrayList { @@ -73,4 +75,25 @@ public void resetToSize(int targetSize) a = new int[targetSize]; size = 0; } + + /** + * Method gratuitously "borrowed" from IntArrayList, would've been nice if that method wasn't private, but ah well. + * + * @param capacity the capacity to grow to + * @see IntArrayList#grow(int) + */ + @SuppressWarnings("ArrayEquality") + private void grow(int capacity) + { + if (capacity <= a.length) { + return; + } + if (a != IntArrays.DEFAULT_EMPTY_ARRAY) { + capacity = (int) Math.max(Math.min((long) a.length + (a.length >> 1), Arrays.MAX_ARRAY_SIZE), capacity); + } else if (capacity < DEFAULT_INITIAL_CAPACITY) { + capacity = DEFAULT_INITIAL_CAPACITY; + } + a = IntArrays.forceCapacity(a, capacity, size); + assert size <= a.length; + } } diff --git a/processing/src/main/java/org/apache/druid/math/expr/Function.java b/processing/src/main/java/org/apache/druid/math/expr/Function.java index 98e12596bac0..d2d9995d9c95 100644 --- a/processing/src/main/java/org/apache/druid/math/expr/Function.java +++ b/processing/src/main/java/org/apache/druid/math/expr/Function.java @@ -21,6 +21,7 @@ import com.google.common.base.Preconditions; import com.google.common.collect.ImmutableSet; +import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet; import it.unimi.dsi.fastutil.objects.ObjectArrays; import it.unimi.dsi.fastutil.objects.ObjectOpenHashSet; import org.apache.druid.annotations.SuppressFBWarnings; @@ -4253,7 +4254,7 @@ public ExprEval apply(List args, Expr.ObjectBinding bindings) } }; } - final Set set = new TreeSet<>(lhsArrayType.getElementType().getNullableStrategy()); + final Set set = new ObjectAVLTreeSet<>(lhsArrayType.getElementType().getNullableStrategy()); set.addAll(Arrays.asList(rhsArray)); return new OverlapConstantArray(set); } diff --git a/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java b/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java index 5d6a0b24fe39..0bb1e28577ad 100644 --- a/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java +++ b/processing/src/main/java/org/apache/druid/segment/ComparatorSortedDimensionDictionary.java @@ -25,7 +25,6 @@ import it.unimi.dsi.fastutil.objects.Object2IntRBTreeMap; import it.unimi.dsi.fastutil.objects.Object2IntSortedMap; -import java.util.ArrayList; import java.util.Comparator; import java.util.List; @@ -44,35 +43,17 @@ public class ComparatorSortedDimensionDictionary public ComparatorSortedDimensionDictionary(List idToValue, Comparator comparator, int length) { - final Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(comparator); - int nullId = -1; + Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(comparator); for (int id = 0; id < length; id++) { - final T value = idToValue.get(id); - if (value == null) { - nullId = id; - } else { - sortedMap.put(value, id); - } + T value = idToValue.get(id); + sortedMap.put(value, id); } this.sortedVals = Lists.newArrayList(sortedMap.keySet()); - int nullIndex = 0; - if (nullId >= 0) { - while (nullIndex < sortedVals.size() && comparator.compare(sortedVals.get(nullIndex), null) < 0) { - nullIndex++; - } - this.sortedVals.add(nullIndex, null); - } this.idToIndex = new int[length]; this.indexToId = new int[length]; - final List sortedIds = new ArrayList<>(sortedMap.size() + (nullId >= 0 ? 1 : 0)); - for (IntIterator iterator = sortedMap.values().iterator(); iterator.hasNext(); ) { - sortedIds.add(iterator.nextInt()); - } - if (nullId >= 0) { - sortedIds.add(nullIndex, nullId); - } int index = 0; - for (final int id : sortedIds) { + for (IntIterator iterator = sortedMap.values().iterator(); iterator.hasNext(); ) { + int id = iterator.nextInt(); idToIndex[id] = index; indexToId[index] = id; index++; diff --git a/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java b/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java index 2f3f57e19f7e..fdd79b341a19 100644 --- a/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java +++ b/processing/src/main/java/org/apache/druid/segment/SortedDimensionDictionary.java @@ -39,30 +39,17 @@ public class SortedDimensionDictionary> public SortedDimensionDictionary(List idToValue, int length) { - final Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(Comparators.naturalNullsFirst()); - int nullId = -1; + Object2IntSortedMap sortedMap = new Object2IntRBTreeMap<>(Comparators.naturalNullsFirst()); for (int id = 0; id < length; id++) { - final T value = idToValue.get(id); - if (value == null) { - nullId = id; - } else { - sortedMap.put(value, id); - } + T value = idToValue.get(id); + sortedMap.put(value, id); } this.sortedVals = Lists.newArrayList(sortedMap.keySet()); - if (nullId >= 0) { - this.sortedVals.add(0, null); - } this.idToIndex = new int[length]; this.indexToId = new int[length]; int index = 0; - if (nullId >= 0) { - idToIndex[nullId] = index; - indexToId[index] = nullId; - index++; - } for (IntIterator iterator = sortedMap.values().iterator(); iterator.hasNext(); ) { - final int id = iterator.nextInt(); + int id = iterator.nextInt(); idToIndex[id] = index; indexToId[index] = id; index++; diff --git a/server/pom.xml b/server/pom.xml index 0e9f1cb79990..42282f03832b 100644 --- a/server/pom.xml +++ b/server/pom.xml @@ -175,7 +175,7 @@ it.unimi.dsi - fastutil + fastutil-core com.github.ben-manes.caffeine diff --git a/sql/pom.xml b/sql/pom.xml index 56707abe63cb..7e342d1e431a 100644 --- a/sql/pom.xml +++ b/sql/pom.xml @@ -161,7 +161,7 @@ it.unimi.dsi - fastutil + fastutil-core com.fasterxml.jackson.core From a542235a1141e4fe93420eddc23908ecd39a80e7 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Thu, 29 Oct 2026 10:13:17 +0800 Subject: [PATCH 21/21] fix: update t-digest intermediate size estimate for 3.3 t-digest 3.3 sizes the MergingDigest temp buffer as 5 * size instead of 5 * compression, so the old estimate undercounted on-heap memory per row. Add a test that checks the estimate against the arrays the library allocates. --- .../tdigestsketch/TDigestSketchUtils.java | 12 ++++-- .../TDigestSketchAggregatorFactoryTest.java | 38 +++++++++++++++++++ 2 files changed, 46 insertions(+), 4 deletions(-) diff --git a/extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchUtils.java b/extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchUtils.java index 8d27659dbc10..efb2ad3cff92 100644 --- a/extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchUtils.java +++ b/extensions-contrib/tdigestsketch/src/main/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchUtils.java @@ -62,15 +62,19 @@ static byte[] toBytes(MergingDigest tDigest) * This method computes an estimate of the max intermediate size of a {@link MergingDigest}. * Since there is no utility available in the T-Digest library to compute this size, * the below code is inspired by looking at - * {@link MergingDigest#MergingDigest(double, int, int)} - * This method is current as of 3.2 version and many need to change in future. + * {@link MergingDigest#MergingDigest(double, int, int)} with default buffer and centroid sizes. + * This method is current as of t-digest 3.3. TDigestSketchAggregatorFactoryTest checks this estimate + * against the arrays the library allocates, so it fails if a future version changes the sizing. */ static int getMaxIntermdiateTDigestSize(int compression) { + // MergingDigest raises compression to at least 10 and adds extra centroid room for small compressions + final int effectiveCompression = Math.max(compression, 10); + final int sizeFudge = effectiveCompression < 30 ? 30 : 10; + final int centroidArraySize = 2 * effectiveCompression + sizeFudge; + final int tempBufferSize = 5 * centroidArraySize; int intermediateSize = 0; - int centroidArraySize = (int) (2 * Math.ceil(compression)) + 10; intermediateSize += 2 * centroidArraySize * Double.BYTES; - int tempBufferSize = (int) (5 * Math.ceil(compression)); intermediateSize += 2 * tempBufferSize * Double.BYTES; intermediateSize += tempBufferSize * Integer.BYTES; // Adding an extra buffer of 1K for overhead diff --git a/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchAggregatorFactoryTest.java b/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchAggregatorFactoryTest.java index 1e618d48ff50..91ad55a179b1 100644 --- a/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchAggregatorFactoryTest.java +++ b/extensions-contrib/tdigestsketch/src/test/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestSketchAggregatorFactoryTest.java @@ -19,6 +19,7 @@ package org.apache.druid.query.aggregation.tdigestsketch; +import com.tdunning.math.stats.MergingDigest; import org.apache.druid.java.util.common.granularity.Granularities; import org.apache.druid.query.Druids; import org.apache.druid.query.aggregation.CountAggregatorFactory; @@ -31,6 +32,10 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.lang.reflect.Array; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; + public class TDigestSketchAggregatorFactoryTest { @Test @@ -86,4 +91,37 @@ public void testCompressionBelowMaxCompressionUnchanged() TDigestSketchAggregatorFactory factory = new TDigestSketchAggregatorFactory("tdigest", "col", 100, config); Assertions.assertEquals(100, factory.getCompression()); } + + /** + * Guards {@link TDigestSketchUtils#getMaxIntermdiateTDigestSize} against changes in the t-digest library. + * The estimate is used for on-heap memory accounting during ingestion, so if it is smaller than what a + * {@link MergingDigest} really allocates, ingestion persists later than maxBytesInMemory intends. + */ + @Test + public void testMaxIntermediateSizeCoversDigestBuffers() throws IllegalAccessException + { + // Raise the cap so the factory doesn't reduce the larger compressions below + final TDigestConfig config = TDigestConfig.builder().maxCompression(1000).build(); + // 5 is raised to the library minimum of 10; 29 and 30 sit on either side of the extra centroid room below 30 + for (int compression : new int[]{5, 10, 29, 30, 100, 200, 1000}) { + final MergingDigest digest = new MergingDigest(compression); + // Sum the sizes of the digest's instance arrays (centroids and temp buffers), which dominate its footprint + long arrayBytes = 0; + for (Field field : MergingDigest.class.getDeclaredFields()) { + if (field.getType().isArray() && !Modifier.isStatic(field.getModifiers())) { + field.setAccessible(true); + final Object array = field.get(digest); + if (array != null) { + final int elementBytes = field.getType().getComponentType() == double.class ? Double.BYTES : Integer.BYTES; + arrayBytes += (long) Array.getLength(array) * elementBytes; + } + } + } + final TDigestSketchAggregatorFactory factory = new TDigestSketchAggregatorFactory("sketch", "col", compression, config); + Assertions.assertTrue( + factory.getMaxIntermediateSize() >= arrayBytes, + "compression " + compression + ": estimate " + factory.getMaxIntermediateSize() + " < allocated " + arrayBytes + ); + } + } }