Skip to content

fix: Validate the first operand of quantile aggregators - #20782

Merged
FrankChen021 merged 7 commits into
apache:masterfrom
havefunatcode:quantile-literal-arg-validation
Nov 4, 2026
Merged

FrankChen021 merged 7 commits into
apache:masterfrom
havefunatcode:quantile-literal-arg-validation

Conversation

@havefunatcode

@havefunatcode havefunatcode commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #18010.

Description

APPROX_QUANTILE('m1', 0.99), with the column name in single quotes, is planned with the string literal as the aggregator input. On master, APPROX_QUANTILE and APPROX_QUANTILE_DS return NaN (or throw ClassCastException when vectorized), and APPROX_QUANTILE_FIXED_BUCKETS fails at runtime with Only serialization version 1 is supported.

The quantile aggregators declare their first operand as ANY, and DefaultOperandTypeChecker doesn't check ANY operands at all. The fix follows the approach @FrankChen021 suggested in the issue.

OperandType in DefaultOperandTypeChecker

Over 50 functions use this checker, so the new behavior is opt-in. An operand can be declared with an OperandType instead of a SqlTypeFamily, through a new operandTypes(OperandType...) overload on the builder (also on OperatorConversions.OperatorBuilder):

.operandTypes(
    OperandType.numeric().orComplex(TDigestSketchAggregatorFactory.TYPE).coerceStrings().nullable(),
    OperandType.of(SqlTypeFamily.NUMERIC)
)
  • orComplex(ColumnType...): complex types the operand also accepts, e.g. the aggregator's own sketch type.
  • coerceStrings(): lets a NUMERIC operand also accept STRING input. The checker adds an explicit CAST to DOUBLE, so the conversion shows up in the plan.
  • nullable(): accepts a NULL literal even if the operand is required. Optional operands already accept it.

For an operand with coerceStrings(), a string literal that can't be parsed as a number is rejected before the cast is added, since the cast would turn it into null:

Function [APPROX_QUANTILE] argument [column] must be numeric, got string literal ['m1']. If you meant to refer to a column, use double quotes.

Numeric strings like '1.5', STRING columns and DATETIME input such as __time still work. Parsing uses ExprEval.computeNumber, the same as at runtime, so 'true' and 'false' are accepted too.

Quantile aggregators

APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS, DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now declare their first operand as a nullable NUMERIC that coerces strings and also accepts their own sketch type. These inputs are now rejected at planning time:

  • BOOLEAN and ARRAY
  • complex types other than the aggregator's own sketch

NULL and CAST(NULL AS DOUBLE) are still accepted. What they return differs between aggregators (NaN for APPROX_QUANTILE, 0.0 for APPROX_QUANTILE_FIXED_BUCKETS), and that's left for a separate discussion.

The Spectator functions still use OperandTypes.ANY.

Aggregations

toDruidExpressionForNumericAggregator also rejects non-numeric string literals for aggregators that still declare ANY. getArgumentsForSimpleAggregator skips the check, because COUNT and SINGLE_VALUE use it and COUNT(DISTINCT 'abc') has to keep working.

SUM('abc') and AVG('abc') still return null. Calcite coerces the literal to a null DECIMAL before Druid sees it. Per the issue, that's left for a separate PR. numeric_agg_string_args.iq records the current behavior.

Release note

APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS, DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now check the type of their first argument. They fail at planning time with a 400 error if the first argument is a string literal that can't be parsed as a number (for example a column name in single quotes), a BOOLEAN, an ARRAY or an unrelated complex type. Previously some of these returned NaN or failed at runtime.


Key changed/added classes in this PR
  • OperandType
  • DefaultOperandTypeChecker
  • OperatorConversions
  • DruidTypeCoercion
  • Calcites
  • Aggregations
  • QuantileSqlAggregator, FixedBucketsHistogramQuantileSqlAggregator, DoublesSketchApproxQuantileSqlAggregator, DoublesSketchObjectSqlAggregator, TDigestSketchQuantileSqlAggregator, TDigestGenerateSketchSqlAggregator

This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

APPROX_QUANTILE('m1', 0.5) with a single-quoted column name was planned
with the string literal as input and returned NaN or failed at runtime.
The quantile aggregators declared their first operand as ANY, and
DefaultOperandTypeChecker doesn't check ANY operands.

DefaultOperandTypeChecker gets two opt-in builder options:
operandComplexTypes lists the complex types an operand accepts, and
coercibleOperands lets a numeric operand take STRING input through an
implicit CAST. A string literal that can't be parsed as a number is
rejected with InvalidSqlInput before the cast is added. TIMESTAMP input
keeps working.

APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS,
DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now
declare their first operand as NUMERIC and allow their own sketch type.
BOOLEAN, ARRAY, other complex types and NULL literals are rejected at
planning time.

toDruidExpressionForNumericAggregator also rejects non-numeric string
literals, for aggregators that still declare ANY.

Fixes apache#18010
@havefunatcode havefunatcode changed the title Validate the first operand of quantile aggregators fix: Validate the first operand of quantile aggregators Oct 29, 2026

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No actionable issues found in this review. The validation changes are consistently applied across the affected quantile aggregators and preserve supported complex, string, and datetime inputs.

Reviewed 19 of 19 changed files.

Validation: targeted SQL tests passed; 31 tests, 0 failures.


This is an automated review by Codex GPT-5.6-Luna(max)

@FrankChen021 FrankChen021 added the Approval Recommended Used when AI completes a review and reports no findings label Oct 29, 2026
Replace the operandComplexTypes() and coercibleOperands() builder
options with an OperandType value class, so each operand is declared
in one place, e.g.

    OperandType.numeric().orComplex(type).coerceStrings()

DefaultOperandTypeChecker.Builder and
OperatorConversions.OperatorBuilder get an operandTypes(OperandType...)
overload, and the SqlTypeFamily overloads delegate to it.
coerceStrings() fails for non-NUMERIC operands.
}

@Test
public void testApproxQuantileWithNonNumericStringLiteral()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add a test against numeric string literal

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added testQuantileOnNumericStringLiteral.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head and all 22 changed files: numeric string literal coverage is present for the doubles-sketch aggregators. No remaining issue from this thread.

}

@Test
public void testQuantileWithNonNumericStringLiteral()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs 3 more test cases: string column, numeric string literal, __time column

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added all three.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head and all 22 changed files: the fixed-buckets tests cover the requested string-column, numeric-string-literal, and __time cases. No remaining issue from this thread.

*/
public OperandType orComplex(final ColumnType... types)
{
return new OperandType(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such API call does not make any sense:

OperandType.of(SqlTypeFamily.ANY).orComplex(DoublesSketchModule.TYPE)

So in this method, we can check this case and throw exception like that in coerceStrings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, it throws for ANY now, like coerceStrings() does for non-NUMERIC.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head and all 22 changed files: orComplex now rejects ANY operands with the requested defensive exception. No remaining issue from this thread.

@FrankChen021 FrankChen021 removed the Approval Recommended Used when AI completes a review and reports no findings label Nov 2, 2026
@FrankChen021

Copy link
Copy Markdown
Member

about the null handling, my suggestion is to keep existing behaviour which accepts NULL as input.
To support it, we can add nullable() method to OperandType so that callers can declare whether nullable is acceptabled.

And we need test cases for both NULL input and CAST(NULL as DOUBLE) for these 6 aggregators, for example:

APPROX_QUANTILE(NULL, 0.5) 
APPROX_QUANTILE(CAST(NULL AS DOUBLE), 0.5) 

Currently APPROX_QUANTILE(NULL, 0.5) returns NaN and APPROX_QUANTILE_FIXED_BUCKETS(NULL, 0.5, 20, 0.0, 10.0) returns 0.0. Whether current returnings are correct and should be aligned, is another thing that can be discussed separately.

Add tests for numeric string literals in APPROX_QUANTILE_DS and for
string columns, numeric string literals and __time in
APPROX_QUANTILE_FIXED_BUCKETS.
@havefunatcode

Copy link
Copy Markdown
Contributor Author

Thanks. Before I change this, could you share a bit more about why? I brought this up in #18010 (comment) (that APPROX_QUANTILE(NULL, 0.5) would start failing with "Illegal use of 'NULL'"), and I took your reply as OK with that, so I kept it. Maybe I misread it.

Only a bare NULL is rejected on this branch. APPROX_QUANTILE_DS(CAST(NULL AS DOUBLE), 0.5) still plans fine.

I don't mind adding nullable() if you'd rather keep the old behavior, I just want to check I'm not missing a case where someone depends on it. If we go that way, I'll add NULL and CAST(NULL AS DOUBLE) tests for all six aggregators. The NaN vs 0.0 difference can be a separate issue.

@FrankChen021

Copy link
Copy Markdown
Member

Thanks. Before I change this, could you share a bit more about why? I brought this up in #18010 (comment) (that APPROX_QUANTILE(NULL, 0.5) would start failing with "Illegal use of 'NULL'"), and I took your reply as OK with that, so I kept it. Maybe I misread it.

Only a bare NULL is rejected on this branch. APPROX_QUANTILE_DS(CAST(NULL AS DOUBLE), 0.5) still plans fine.

I don't mind adding nullable() if you'd rather keep the old behavior, I just want to check I'm not missing a case where someone depends on it. If we go that way, I'll add NULL and CAST(NULL AS DOUBLE) tests for all six aggregators. The NaN vs 0.0 difference can be a separate issue.

The reason is I found different aggregators have different implementation on how to handle the NULL input. And existing spectator DOES have a test case against NULL input and expect NULL returning ( see testSpectatorFunctionsOnNullHistogram). That one is similar to above quantile functions you change in this PR.

Although we don't have a general rule about handling of NULL, it's hard to tell which behaviour is good/right/better, I think it's better for us keep existing behaviour in this RP and discuss the returning separately.

Add OperandType.nullable() so that a required operand can accept a NULL
literal, and use it for the six quantile aggregators. Add tests for NULL
and CAST(NULL AS DOUBLE) inputs.
@havefunatcode

Copy link
Copy Markdown
Contributor Author

Got it, thanks for explaining. Done in the latest commit. OperandType has nullable() now and the six aggregators use it, so NULL is accepted again. I added NULL and CAST(NULL AS DOUBLE) tests for each of them, and updated the description.

Something I ran into while writing the TDigest tests: TDIGEST_QUANTILE(m1, 0.5) and TDIGEST_GENERATE_SKETCH(m1) in the same query fail with Duplicate field name: a0:agg. It happens with a regular column too, so it's not about NULL. I split the tests for now. I can open a separate issue for it if that's useful.

final IntSet nullableOperands = requiredOperandCount == null
? new IntArraySet()
: DefaultOperandTypeChecker.buildNullableOperands(requiredOperandCount, operandTypes.size(), new IntArraySet());
if (operandTypes != null) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

operandTypes can't be null here, right? and this new if block should be moved into buildNullableOperands

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be null when the operator is built with operandTypeChecker instead of operandTypes, like GREATEST and LEAST. Without the check, CalciteQueryTest.testGreatestLeastTypes failed with an NPE, so I kept the check and added a comment explaining why.

Moved the rest into buildNullableOperands. It takes the operand types now, so DefaultOperandTypeChecker doesn't need the separate isNullable() check either.

I also added the missing tests for the other aggregators, so they all use the same inputs now (numeric strings, __time, and BOOLEAN/ARRAY/complex for DS_QUANTILES_SKETCH and TDIGEST_GENERATE_SKETCH).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head and all 22 changed files: custom checker operators remain null-safe, nullable handling is centralized in buildNullableOperands, and the regression coverage is present. No remaining issue from this thread.

@FrankChen021

Copy link
Copy Markdown
Member

Something I ran into while writing the TDigest tests: TDIGEST_QUANTILE(m1, 0.5) and TDIGEST_GENERATE_SKETCH(m1) in the same query fail with Duplicate field name: a0:agg. It happens with a regular column too, so it's not about NULL. I split the tests for now. I can open a separate issue for it if that's useful.

Let's create a new issue to track this. and if you are willing to solve it, a PR is welcomed.

buildNullableOperands takes the operand types now and includes operands
declared with OperandType.nullable(). Add numeric string literal and __time
tests for APPROX_QUANTILE and TDIGEST_QUANTILE, and the same inputs plus
BOOLEAN, ARRAY and complex inputs for DS_QUANTILES_SKETCH and
TDIGEST_GENERATE_SKETCH.
@havefunatcode

Copy link
Copy Markdown
Contributor Author

Opened #20819 for it.

master upgraded t-digest to 3.3 in apache#20765, which changes the median of
__time in testQuantileAndSketchOnTimeColumn. The sketch is the same.
@havefunatcode

Copy link
Copy Markdown
Contributor Author

CI failed on testQuantileAndSketchOnTimeColumn. My branch was still on t-digest 3.2, and master moved to 3.3 in #20765, which gives a different median for __time. I updated the expected value in the latest commit and it passes locally with 3.3. The other failure is IndexParallelTaskTest, which looks like the flaky test in #20809.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The current head resolves all five supplied follow-up threads: typed operand declarations, ANY misuse rejection, numeric-string coverage, fixed-bucket coverage, and null-safe custom-checker inference. The complete current-head diff and surrounding code were reviewed with no actionable issue found.

Reviewed 22 of 22 changed files.

Validation: git -C /Users/frankchen/.codex/mergelens/worktrees/apache__druid__pr-20782 diff --check a311ccd1c4ffdc2073c1b054dd57378e102dfed9 be10a032df06a201c82197d29d0e928c437f863e -- passed with no output.


This is an automated review by Codex GPT-5.6-Luna(max)

@FrankChen021 FrankChen021 added Approval Recommended Used when AI completes a review and reports no findings Upgrade note Behavior change that requires an upgrade note labels Nov 2, 2026
@FrankChen021
FrankChen021 merged commit edcbfe7 into apache:master Nov 4, 2026
27 checks passed
@github-actions github-actions Bot added this to the 40.0.0 milestone Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approval Recommended Used when AI completes a review and reports no findings Area - Querying Upgrade note Behavior change that requires an upgrade note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check the types of given parameters matches the types of SQL function declared parameters

3 participants