Repository navigation
fix: Validate the first operand of quantile aggregators - #20782
Conversation
APPROX_QUANTILE('m1', 0.5) with a single-quoted column name was planned
with the string literal as input and returned NaN or failed at runtime.
The quantile aggregators declared their first operand as ANY, and
DefaultOperandTypeChecker doesn't check ANY operands.
DefaultOperandTypeChecker gets two opt-in builder options:
operandComplexTypes lists the complex types an operand accepts, and
coercibleOperands lets a numeric operand take STRING input through an
implicit CAST. A string literal that can't be parsed as a number is
rejected with InvalidSqlInput before the cast is added. TIMESTAMP input
keeps working.
APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS,
DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now
declare their first operand as NUMERIC and allow their own sketch type.
BOOLEAN, ARRAY, other complex types and NULL literals are rejected at
planning time.
toDruidExpressionForNumericAggregator also rejects non-numeric string
literals, for aggregators that still declare ANY.
Fixes apache#18010
FrankChen021
left a comment
There was a problem hiding this comment.
🟢 Approval recommended
No actionable issues found in this review. The validation changes are consistently applied across the affected quantile aggregators and preserve supported complex, string, and datetime inputs.
Reviewed 19 of 19 changed files.
Validation: targeted SQL tests passed; 31 tests, 0 failures.
This is an automated review by Codex GPT-5.6-Luna(max)
Replace the operandComplexTypes() and coercibleOperands() builder
options with an OperandType value class, so each operand is declared
in one place, e.g.
OperandType.numeric().orComplex(type).coerceStrings()
DefaultOperandTypeChecker.Builder and
OperatorConversions.OperatorBuilder get an operandTypes(OperandType...)
overload, and the SqlTypeFamily overloads delegate to it.
coerceStrings() fails for non-NUMERIC operands.
| } | ||
|
|
||
| @Test | ||
| public void testApproxQuantileWithNonNumericStringLiteral() |
There was a problem hiding this comment.
let's add a test against numeric string literal
There was a problem hiding this comment.
Added testQuantileOnNumericStringLiteral.
There was a problem hiding this comment.
Rechecked the current head and all 22 changed files: numeric string literal coverage is present for the doubles-sketch aggregators. No remaining issue from this thread.
| } | ||
|
|
||
| @Test | ||
| public void testQuantileWithNonNumericStringLiteral() |
There was a problem hiding this comment.
needs 3 more test cases: string column, numeric string literal, __time column
There was a problem hiding this comment.
Added all three.
There was a problem hiding this comment.
Rechecked the current head and all 22 changed files: the fixed-buckets tests cover the requested string-column, numeric-string-literal, and __time cases. No remaining issue from this thread.
| */ | ||
| public OperandType orComplex(final ColumnType... types) | ||
| { | ||
| return new OperandType( |
There was a problem hiding this comment.
Such API call does not make any sense:
OperandType.of(SqlTypeFamily.ANY).orComplex(DoublesSketchModule.TYPE)So in this method, we can check this case and throw exception like that in coerceStrings
There was a problem hiding this comment.
Done, it throws for ANY now, like coerceStrings() does for non-NUMERIC.
There was a problem hiding this comment.
Rechecked the current head and all 22 changed files: orComplex now rejects ANY operands with the requested defensive exception. No remaining issue from this thread.
|
about the null handling, my suggestion is to keep existing behaviour which accepts NULL as input. And we need test cases for both APPROX_QUANTILE(NULL, 0.5)
APPROX_QUANTILE(CAST(NULL AS DOUBLE), 0.5) Currently |
Add tests for numeric string literals in APPROX_QUANTILE_DS and for string columns, numeric string literals and __time in APPROX_QUANTILE_FIXED_BUCKETS.
|
Thanks. Before I change this, could you share a bit more about why? I brought this up in #18010 (comment) (that Only a bare I don't mind adding |
The reason is I found different aggregators have different implementation on how to handle the Although we don't have a general rule about handling of NULL, it's hard to tell which behaviour is good/right/better, I think it's better for us keep existing behaviour in this RP and discuss the returning separately. |
Add OperandType.nullable() so that a required operand can accept a NULL literal, and use it for the six quantile aggregators. Add tests for NULL and CAST(NULL AS DOUBLE) inputs.
|
Got it, thanks for explaining. Done in the latest commit. Something I ran into while writing the TDigest tests: |
| final IntSet nullableOperands = requiredOperandCount == null | ||
| ? new IntArraySet() | ||
| : DefaultOperandTypeChecker.buildNullableOperands(requiredOperandCount, operandTypes.size(), new IntArraySet()); | ||
| if (operandTypes != null) { |
There was a problem hiding this comment.
operandTypes can't be null here, right? and this new if block should be moved into buildNullableOperands
There was a problem hiding this comment.
It can be null when the operator is built with operandTypeChecker instead of operandTypes, like GREATEST and LEAST. Without the check, CalciteQueryTest.testGreatestLeastTypes failed with an NPE, so I kept the check and added a comment explaining why.
Moved the rest into buildNullableOperands. It takes the operand types now, so DefaultOperandTypeChecker doesn't need the separate isNullable() check either.
I also added the missing tests for the other aggregators, so they all use the same inputs now (numeric strings, __time, and BOOLEAN/ARRAY/complex for DS_QUANTILES_SKETCH and TDIGEST_GENERATE_SKETCH).
There was a problem hiding this comment.
Rechecked the current head and all 22 changed files: custom checker operators remain null-safe, nullable handling is centralized in buildNullableOperands, and the regression coverage is present. No remaining issue from this thread.
Let's create a new issue to track this. and if you are willing to solve it, a PR is welcomed. |
buildNullableOperands takes the operand types now and includes operands declared with OperandType.nullable(). Add numeric string literal and __time tests for APPROX_QUANTILE and TDIGEST_QUANTILE, and the same inputs plus BOOLEAN, ARRAY and complex inputs for DS_QUANTILES_SKETCH and TDIGEST_GENERATE_SKETCH.
|
Opened #20819 for it. |
master upgraded t-digest to 3.3 in apache#20765, which changes the median of __time in testQuantileAndSketchOnTimeColumn. The sketch is the same.
|
CI failed on |
FrankChen021
left a comment
There was a problem hiding this comment.
🟢 Approval recommended
The current head resolves all five supplied follow-up threads: typed operand declarations, ANY misuse rejection, numeric-string coverage, fixed-bucket coverage, and null-safe custom-checker inference. The complete current-head diff and surrounding code were reviewed with no actionable issue found.
Reviewed 22 of 22 changed files.
Validation: git -C /Users/frankchen/.codex/mergelens/worktrees/apache__druid__pr-20782 diff --check a311ccd1c4ffdc2073c1b054dd57378e102dfed9 be10a032df06a201c82197d29d0e928c437f863e -- passed with no output.
This is an automated review by Codex GPT-5.6-Luna(max)
Fixes #18010.
Description
APPROX_QUANTILE('m1', 0.99), with the column name in single quotes, is planned with the string literal as the aggregator input. On master, APPROX_QUANTILE and APPROX_QUANTILE_DS return NaN (or throwClassCastExceptionwhen vectorized), and APPROX_QUANTILE_FIXED_BUCKETS fails at runtime withOnly serialization version 1 is supported.The quantile aggregators declare their first operand as
ANY, andDefaultOperandTypeCheckerdoesn't checkANYoperands at all. The fix follows the approach @FrankChen021 suggested in the issue.OperandTypeinDefaultOperandTypeCheckerOver 50 functions use this checker, so the new behavior is opt-in. An operand can be declared with an
OperandTypeinstead of aSqlTypeFamily, through a newoperandTypes(OperandType...)overload on the builder (also onOperatorConversions.OperatorBuilder):orComplex(ColumnType...): complex types the operand also accepts, e.g. the aggregator's own sketch type.coerceStrings(): lets a NUMERIC operand also accept STRING input. The checker adds an explicit CAST to DOUBLE, so the conversion shows up in the plan.nullable(): accepts a NULL literal even if the operand is required. Optional operands already accept it.For an operand with
coerceStrings(), a string literal that can't be parsed as a number is rejected before the cast is added, since the cast would turn it into null:Numeric strings like
'1.5', STRING columns and DATETIME input such as__timestill work. Parsing usesExprEval.computeNumber, the same as at runtime, so'true'and'false'are accepted too.Quantile aggregators
APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS, DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now declare their first operand as a nullable NUMERIC that coerces strings and also accepts their own sketch type. These inputs are now rejected at planning time:
NULLandCAST(NULL AS DOUBLE)are still accepted. What they return differs between aggregators (NaN for APPROX_QUANTILE, 0.0 for APPROX_QUANTILE_FIXED_BUCKETS), and that's left for a separate discussion.The Spectator functions still use
OperandTypes.ANY.AggregationstoDruidExpressionForNumericAggregatoralso rejects non-numeric string literals for aggregators that still declareANY.getArgumentsForSimpleAggregatorskips the check, because COUNT and SINGLE_VALUE use it andCOUNT(DISTINCT 'abc')has to keep working.SUM('abc')andAVG('abc')still return null. Calcite coerces the literal to a null DECIMAL before Druid sees it. Per the issue, that's left for a separate PR.numeric_agg_string_args.iqrecords the current behavior.Release note
APPROX_QUANTILE, APPROX_QUANTILE_FIXED_BUCKETS, APPROX_QUANTILE_DS, DS_QUANTILES_SKETCH, TDIGEST_QUANTILE and TDIGEST_GENERATE_SKETCH now check the type of their first argument. They fail at planning time with a 400 error if the first argument is a string literal that can't be parsed as a number (for example a column name in single quotes), a BOOLEAN, an ARRAY or an unrelated complex type. Previously some of these returned NaN or failed at runtime.
Key changed/added classes in this PR
OperandTypeDefaultOperandTypeCheckerOperatorConversionsDruidTypeCoercionCalcitesAggregationsQuantileSqlAggregator,FixedBucketsHistogramQuantileSqlAggregator,DoublesSketchApproxQuantileSqlAggregator,DoublesSketchObjectSqlAggregator,TDigestSketchQuantileSqlAggregator,TDigestGenerateSketchSqlAggregatorThis PR has: