Repository navigation
DRILL-8555: Physical plan cache for parameterized SQL queries - #3086
letian-jiang wants to merge 16 commits into
Conversation
|
@shfshihuafeng Thank you for this PR. Did you take a look at #3023? I realize they are somewhat different, but it seems the overall goal is similar in trying to cache plans. Are there components of that PR that you could incorporate in yours? |
|
@cgivre Thanks for pointing me to #3023. I have reviewed it, and I agree that both PRs share the same underlying goal: reusing planning work to reduce the overhead of repeated queries. Before considering #3023 production-ready, I think we would need to establish the conditions for safe plan reuse, even for identical SQL. For example:
There is also a practical consideration: production workloads often repeat the same query shape with different literals. Restricting reuse to identical SQL text would substantially limit the benefit for those workloads. My implementation focuses on these correctness requirements through eligibility checks, context and table compatibility checks, and reconstruction of a fresh plan and scan state for each execution. It also preserves typed parameter slots so queries with different literals can reuse compatible cached plans. |
cgivre
left a comment
There was a problem hiding this comment.
A few issues with parameterized planning and the BIGINT literal change; details inline.
| templateTextPlan, cacheReader, cacheContext)); | ||
| } | ||
| return planned; | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
This also swallows genuine planning failures from handler.getPlan(candidate.sql) (validation errors, planner timeouts), so failing queries are planned twice. The retry reuses sqlNode subtrees that the first validation already rewrote in place.
Fix: catch only cache-specific failures here (or rethrow ValidationException/UserException/planner timeouts).
There was a problem hiding this comment.
Agreed. I moved handler.getPlan(...) outside the cache fallback block, so cache lookup or binding failures can fall back before planning, while planning failures propagate without a cache-induced retry.
| } | ||
| if (lExpr.getDynamicParamIndex() < 0) { | ||
| // Small BIGINT values otherwise parse back as INT during a plan round trip. | ||
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); |
There was a problem hiding this comment.
This applies to all plans, not just cached ones: after a JSON round trip, BIGINT literals come back as CastExpression. DrillExprToPaimonTranslator has no visitCastExpression, so in multi-fragment plans the Paimon predicate is dropped after Drill has already removed the Filter, and bigint_col = 5 returns unfiltered rows. Only the Iceberg translator was updated.
Fix: either unwrap cast-of-literal in the pushdown translators (Paimon and any others), or have the parser read the round-tripped value back as a BIGINT literal instead of emitting a cast.
There was a problem hiding this comment.
Thanks for identifying the impact beyond cached plans. I addressed this in the expression parser: the BIGINT cast emitted when serializing an ordinary integer literal is now restored as a LongExpression during deserialization. Explicit casts involving parameter slots remain intact.
c74a235 to
f5e64d5
Compare
|
Hi @cgivre, thanks again for your review! I’ve pushed updates addressing your feedback on parameterized planning and BIGINT literal round trips. When you have a chance, could you take another look? Please let me know if there’s anything else you’d like me to address. Thanks! |
|
Suggestion: make the plan cache size and expiration configurable
Could these be boot options in A few related points:
|
|
Question: plans to support the other storage plugins? Right now only HBase and Iceberg opt in ( I understand why they can't simply be switched on. On a hit, the cache rebinds literals in Drill expressions. But most plugins turn those literals into native state during planning and keep only the result:
Reusing those plans without rebuilding that state would silently return wrong results. So each plugin needs to keep the source expression in its scan JSON and redo pushdown/pruning on bind, as A possible path, roughly in order of effort:
It would be worth measuring planning time against bind-time rebuild cost for Given how large this PR already is, I'm fine with plugin expansion coming in follow-ups. But I'd like to understand the roadmap, and to make sure the plugin API in this PR makes opting in as cheap as possible. Could you add the plan (or JIRAs for each plugin) to the design doc? |
|
Review findings Line numbers refer to the PR head. The first three are correctness issues I'd like to see fixed before merge. Correctness
Performance / design
Observability and operations
Docs
|
|
@cgivre Thank you for the detailed and thoughtful review! I’ve learned a lot from your comments. |
I’ve made the cache size and both expiration policies configurable.
|
cgivre
left a comment
There was a problem hiding this comment.
Line-by-line follow-up to my earlier review comment, with suggested changes where there's a concrete fix.
I applied all the suggestions together on top of 53306c1, and exec/java-exec (with its dependencies, including logical) builds cleanly, checkstyle included. Several depend on each other, so it's easiest to add them to a batch and commit them together:
- the
PlanCacheconstructor, its three imports, andnew PlanCache(config)inDrillbitContext writeAfterSuccess/putinPlanCacheand the serialization change inDrillSqlWorkerkeyFingerprint()inPlanCacheand the key inDrillSqlWorker
The drill-module.conf defaults are in the comment on the PlanCache fields, because the right spot in that file is outside the diff.
Tests: I couldn't find any plan-cache tests in this PR. PLAN_CACHE_PLUGIN_GUIDE.md describes the checks a plugin should pass (results with caching off vs. a hit with different literals, schema changes, plugin config changes, joins with unsupported plugins, etc.), but none are included for HBase, Iceberg or the engine. Could you add them? They should cover at least:
- the fallback path
- HAVING with GROUP BY
- two sessions with different options sharing a template
- a stale table version
- the new boot options
| // Plan exactly once, outside cache fallback. Validation can mutate the SQL tree, | ||
| // so retrying a failed planning attempt could reuse partially rewritten nodes. | ||
| PhysicalPlan planned = handler.getPlan(planningSql); | ||
| if (prepareCacheInsert != null) { | ||
| prepareCacheInsert.accept(planned); | ||
| } | ||
| return planned; |
There was a problem hiding this comment.
[Correctness] Fall back to ordinary planning when the parameterized SQL fails to plan.
On a miss, the query is planned once from candidate.sql, and the try/catch above only covers lookup and preparation. Any query that only plans with real literals fails, but only when the cache is enabled. Examples: JOIN ... ON TRUE (becomes ON ?), PERCENTILE_CONT(0.5) WITHIN GROUP, and functions that need a literal operand but aren't in configurationOperand.
The comment above is right that the mutated tree can't be retried. This suggestion avoids that by re-entering getQueryPlan from the SQL string with planner.enable_plan_cache set to false at query level, the same pattern the Metastore retry in getPlan uses. The fresh parse, converter and handler start clean. A query that fails for a real reason (bad column, etc.) fails again on the second attempt and reports the error from the original SQL.
| // Plan exactly once, outside cache fallback. Validation can mutate the SQL tree, | |
| // so retrying a failed planning attempt could reuse partially rewritten nodes. | |
| PhysicalPlan planned = handler.getPlan(planningSql); | |
| if (prepareCacheInsert != null) { | |
| prepareCacheInsert.accept(planned); | |
| } | |
| return planned; | |
| if (prepareCacheInsert == null) { | |
| return handler.getPlan(planningSql); | |
| } | |
| PhysicalPlan planned; | |
| try { | |
| planned = handler.getPlan(planningSql); | |
| } catch (Exception e) { | |
| // Some queries only plan with real literal values (e.g. ON TRUE, or functions that | |
| // need a literal operand). Validation mutates the SQL tree, so re-parse and plan the | |
| // original SQL with the cache off for this query, like the Metastore retry above. | |
| logger.debug("Parameterized planning failed; replanning without the plan cache", e); | |
| context.getOptions().setLocalOption(PlannerSettings.ENABLE_PLAN_CACHE_OPTION, false); | |
| return getQueryPlan(context, sql, textPlan); | |
| } | |
| prepareCacheInsert.accept(planned); | |
| return planned; |
| select.getGroup(), visitNullable(select.getHaving()), | ||
| select.getWindowList(), visitNullable(select.getQualify()), |
There was a problem hiding this comment.
[Correctness] Keep HAVING and QUALIFY verbatim when GROUP BY / the select list are.
When structuralProjection is true, GROUP BY and the select list keep their literals, but HAVING and QUALIFY still get slots. So they no longer match the grouped or windowed expressions:
SELECT x || 'a', COUNT(*) FROM hbase.t GROUP BY x || 'a' HAVING x || 'a' = 'ba'HAVING becomes x || ? = ?, and Calcite raises "Expression 'x' is not being grouped". Without a fallback, the query fails only when the cache is on. Using the same condition for HAVING and QUALIFY keeps them consistent with what they have to match.
| select.getGroup(), visitNullable(select.getHaving()), | |
| select.getWindowList(), visitNullable(select.getQualify()), | |
| select.getGroup(), | |
| structuralProjection ? select.getHaving() : visitNullable(select.getHaving()), | |
| select.getWindowList(), | |
| structuralProjection ? select.getQualify() : visitNullable(select.getQualify()), |
| if (!lExpr.isDynamicParam()) { | ||
| // Preserve integer width in JSON; the parser restores this as a BIGINT literal. | ||
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); | ||
| } else { | ||
| sb.append(lExpr.getLong()); | ||
| } |
There was a problem hiding this comment.
[Correctness / compatibility] Narrow the BIGINT cast to values that need it.
This now writes cast(N as BIGINT) for every non-parameter BIGINT literal in every plan, even with the cache off. That changes EXPLAIN output, scan digests and filter strings (e.g. AbstractGroupScanWithMetadata.getFilterString) for all users.
Only values in int range are ambiguous on read-back, because ValueExpressions.getNumericExpression tries Integer.parseInt first and then Long.parseLong. Values outside int range already round-trip as LongExpression, so they don't need the cast. This suggestion limits the cast to the ambiguous case. The createCast special case in ExprParser.g4 still handles it.
Is the cast needed at all outside the plan cache? Physical plans were already serialized to JSON for remote fragments before this PR. If only cached plans need it, could it be applied only when serializing for the cache?
| if (!lExpr.isDynamicParam()) { | |
| // Preserve integer width in JSON; the parser restores this as a BIGINT literal. | |
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); | |
| } else { | |
| sb.append(lExpr.getLong()); | |
| } | |
| long value = lExpr.getLong(); | |
| if (!lExpr.isDynamicParam() && value >= Integer.MIN_VALUE && value <= Integer.MAX_VALUE) { | |
| // Only int-range values are ambiguous: the parser reads them back as INT. | |
| sb.append("cast(").append(value).append(" as BIGINT)"); | |
| } else { | |
| sb.append(value); | |
| } |
| this.optionsFingerprint = Objects.requireNonNull(optionsFingerprint, "optionsFingerprint"); | ||
| this.tableVersions = Collections.unmodifiableMap(tableVersions); | ||
| this.pluginConfigs = Collections.unmodifiableMap(pluginConfigs); | ||
| } |
There was a problem hiding this comment.
[Performance] Put the options and plugin-config fingerprints in the key (1 of 2; the other suggestion is in DrillSqlWorker).
Today, a mismatch in matchesContext invalidates the shared entry. Two sessions running the same template with different options (e.g. ALTER SESSION SET planner.slice_target = ...) keep evicting each other's entry, so the hit rate is 0 and every query also pays to republish. With the fingerprints in the key, each option set gets its own entry. Only a table-version change, which should invalidate, falls through to invalidate.
| } | |
| } | |
| /** | |
| * Session- and plugin-dependent part of the context. It belongs in the cache key, so | |
| * sessions with different options keep separate entries instead of evicting each other. | |
| * Table versions stay in {@link #matches}, where a change should invalidate the entry. | |
| */ | |
| String keyFingerprint() { | |
| return optionsFingerprint + '\n' + pluginConfigs; | |
| } |
| String key = context.getQueryUserName() + '\n' | ||
| + context.getSession().getDefaultSchemaPath() + '\n' | ||
| + candidate.template; |
There was a problem hiding this comment.
[Performance] Include the session/plugin fingerprint in the key (2 of 2; see ContextSnapshot.keyFingerprint() in PlanCache). This keeps sessions with different option values from evicting each other's entries.
| String key = context.getQueryUserName() + '\n' | |
| + context.getSession().getDefaultSchemaPath() + '\n' | |
| + candidate.template; | |
| String key = context.getQueryUserName() + '\n' | |
| + context.getSession().getDefaultSchemaPath() + '\n' | |
| + snapshot.keyFingerprint() + '\n' | |
| + candidate.template; |
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; |
There was a problem hiding this comment.
Import for the plan-cache metrics.
| import com.fasterxml.jackson.databind.JsonNode; | |
| import com.codahale.metrics.Gauge; | |
| import com.fasterxml.jackson.databind.JsonNode; |
| ExecConstants.STORAGE_PLUGIN_REGISTRY_IMPL, StoragePluginRegistry.class, this); | ||
|
|
||
| reader = new PhysicalPlanReader(config, classpathScan, lpPersistence, endpoint, storagePlugins); | ||
| planCache = new PlanCache(); |
There was a problem hiding this comment.
Pass the boot config so the cache size and expiry can be configured (see the suggestion on PlanCache's fields).
| planCache = new PlanCache(); | |
| planCache = new PlanCache(config); |
| return null; | ||
| } | ||
| String identifier = ((HBaseScanSpec) selection).getTableName(); | ||
| try (Admin admin = getConnection().getAdmin()) { |
There was a problem hiding this comment.
[Performance] Remote metadata I/O on every lookup, including hits.
ContextSnapshot.resolve calls this before every lookup. Each call opens an Admin and makes two master RPCs (tableExists + getTableDescriptor). Iceberg is similar: IcebergFormatPlugin.planCacheTable runs HadoopTables.load, and IcebergGroupScan loads the table again during bind. A hit can end up costing more than the planning it skips.
Possible fixes:
- Drop
tableExistsand callgetDescriptordirectly (it throwsTableNotFoundException, which can map tonull). That's one RPC instead of two. - Use
Connection.getTable(name).getDescriptor()rather than opening anAdminper query. - Cache the descriptor hash briefly (a few seconds) per Drillbit, accepting a short window where a schema change is caught at bind time instead.
Could you include hit latency against full planning time for an HBase point lookup in the PR benchmarks?
| // Keep this list aligned with the value-dependent rewrites in DrillOptiq | ||
| // and PreProcessLogicalRel; ordinary data operands still get slots. | ||
| switch (call.getOperator().getName().toUpperCase(Locale.ROOT)) { | ||
| case "DATE_PART": |
There was a problem hiding this comment.
[Robustness] Hard-coded list of functions whose operands must stay literal.
Any function not on this list that needs a literal operand during validation, return-type inference or DrillOptiq conversion silently gets a slot and fails planning. That includes format strings, scale arguments, percentile fractions and new UDFs. The fallback suggested in DrillSqlWorker stops these from failing queries. It would still be better not to rely on the list: could literal-only operands come from the operator's operand type checker (e.g. SqlOperandTypeChecker / OperandTypes.LITERAL), or from an annotation on Drill UDFs, rather than a name switch?
| @@ -0,0 +1,103 @@ | |||
| <!-- | |||
There was a problem hiding this comment.
[Docs] Please move PLAN_CACHE_DESIGN.md and PLAN_CACHE_PLUGIN_GUIDE.md from the repo root to docs/dev/, and link them from docs/dev/DevDocs.md. It would also help to document the new boot options and metrics, and the plan for supporting other plugins (see my earlier comment).
|
I've updated the parameterizer to preserve HAVING and QUALIFY expressions whenever the SELECT expressions are preserved for grouping, ordering, or window definitions. This keeps matching expressions consistent across clauses. Eligible WHERE literals can still be rebound. SELECT x + 1 AS k,
ROW_NUMBER() OVER (ORDER BY x + 1) AS rn
FROM (VALUES (1, 1), (1, 2), (2, 3)) AS t(x, v)
WHERE v > 0
QUALIFY x + 1 = 2
AND ROW_NUMBER() OVER (ORDER BY x + 1) = 1
ORDER BY x + 1;With plan caching disabled, this returned one row: (k = 2, rn = 1). With caching enabled, it returned zero rows without any exception, both on initial planning and on a confirmed cache hit. An exception-triggered fallback would not catch this regression. |
This is a difficult problem, and I agree that a hard-coded list carries a maintenance risk. Given Drill’s current architecture, though, I don’t see a clearly better practical approach: DrillOptiq itself uses function-name switches for value-dependent rewrites, so the parameterizer needs explicit knowledge of which operands must remain literals. Moving this information into function metadata could improve organization, but would still require manually declaring and maintaining those rules. For now, I think keeping this list aligned with the existing rewrites and adding targeted regression tests is a reasonable approach, while acknowledging that it does not establish safety for every SQL pattern. |
|
Following up on the remaining points:
|
|
I’ve rerun both benchmarks on the latest commit, [6ee1c592b](letian-jiang@6ee1c59), using the same machine, datasets, queries and sampling method as the original measurements.
|
DRILL-8555: Physical plan cache for parameterized SQL queries
Description
This PR adds a Drillbit-scoped physical plan cache with HBase and Iceberg support, reusing plans across connections to reduce repeated validation and optimization. It is disabled by default; enable it with
ALTER SESSION SET planner.enable_plan_cache = true.Method
flowchart LR SQL[SQL] --> Template[SQL template] Template --> Cache{Plan cache} Cache -->|Hit| Bind[Bind literals] Cache -->|Miss| Planner[Plan query] Bind --> Plan[Physical plan] Planner --> PlanEligible literals become parameter slots in the SQL template. A hit binds current values to a fresh copy of the cached plan; a miss plans the query normally and populates the cache after successful execution.
Safety guarantees
Benchmark
Measured on one local Drillbit with a Ryzen 7 9700X, 30 GiB RAM and OpenJDK 21. HBase used a 1,000-row mini-cluster for point reads, 50-row range scans and column filters. Iceberg ran all 22 TPC-H queries over eight SF0.01 tables (Q15 used a derived table; Q19 exposed the common equijoin).
Cache hits
The benefit is largest when planning dominates latency: it accounts for roughly 73–76% of the reported HBase baseline latency, and hits reduce end-to-end latency by 50–56%. Analytical queries also benefit: Iceberg planning drops 83.6%, reducing aggregate end-to-end latency by 10.2%.
HBase values are medians of three run medians (15 pairs per workload per run). Iceberg values are sums of per-query medians (three pairs per query), not suite wall time. Percentages are latency reductions relative to cache-off execution.
Cache misses
A separate warmed comparison cleared the plan cache before each cache-on query and drained background writes before both modes. Misses added a median 3–5 ms of paired end-to-end latency for HBase (45 pairs per workload). For Iceberg, the sum of query end-to-end medians changed from 20,624 to 21,806 ms (+5.7%). Miss overhead was modest in these local measurements, while hits provided the largest benefit for short queries.
Documentation
PLAN_CACHE_DESIGN.md: basic principles and supported scope.PLAN_CACHE_PLUGIN_GUIDE.md: plugin APIs and scan reconstruction requirements.