Repository navigation
Optionally share CTE references that have no quals to push down - #2361
Open
chenjinbao1989 wants to merge 2 commits into
Open
chenjinbao1989 wants to merge 2 commits into
chenjinbao1989 wants to merge 2 commits into
Conversation
A CTE referenced more than once is inlined by default: each reference is planned as a separate subquery, with its own quals pushed down into it. With gp_cte_sharing, all references share a single computation instead, but then none of their quals can be pushed down. Which is better depends on the query. At TPC-DS scale factor 10, with the Postgres planner, gp_cte_sharing makes q23 1.8x faster, where both references are "x IN (SELECT ... FROM cte)", but q04 5.9x, q11 6.0x and q31 6.2x slower. In q31 each of three references to a CTE filters on a different quarter: inlined, each copy pushes its filter down and the partition selectors skip 420 partition scans; shared, the CTE computes all quarters and years, scanning 6.6 times as many store_sales rows. Over all 99 queries, turning on gp_cte_sharing makes the total 15% slower. Add gp_cte_sharing_auto (off by default; only used when gp_cte_sharing is off). It decides per reference: a reference that has quals that can be pushed down into the CTE is inlined, as without sharing, and the others share a single computation of the CTE. The new cte_ref_has_pushable_quals() makes the same checks as push_down_restrict(), without pushing anything down, as push_down_restrict() removes the quals it pushes from the rel's restrictions. The CTE is shared only if its planned result is partitioned across the segments. That is where sharing pays off, and otherwise a reference executed in the QD and one executed in the segments may need the same Shared Scan, which fails with "cannot share ShareInputScan between QD and primary reader/write gang"; gp_cte_sharing fails like that, for example, in gp_stat_reset_single_table_counters(). To check, the shared plan is made before the decision; plan_shared_cte() is split out of set_cte_pathlist() for that. Like gp_cte_sharing, the option is turned off in subqueries planned at lower levels, to avoid deadlocks between Shared Scans, and doesn't apply to recursive CTEs, CTEs that aren't SELECTs, or CTEs with volatile functions. MATERIALIZED and NOT MATERIALIZED still take precedence. At scale factor 10, 8 of the 99 TPC-DS queries change plan, none gets slower, and the total goes from 302.3 s to 275.4 s; q23 from 45.7 s to 25.3 s, q14 from 26.3 s to 22.5 s, q59 from 4.0 s to 3.0 s. The queries that gp_cte_sharing makes slower don't change. The results are unchanged. This is a heuristic. Where all references filter the CTE, they keep the inlined plans, even if sharing would pay off anyway: gp_cte_sharing makes q74 twice as fast, gp_cte_sharing_auto leaves it as it is.
Drop gp_cte_sharing_auto: when gp_cte_sharing is off, the planner now always decides per reference whether to share a CTE. A reference with quals that can be pushed down into the CTE is still inlined, so the queries that gp_cte_sharing makes slower keep their plans. The per-level switch in PlannerConfig is renamed cte_sharing_auto; it is still turned off in subqueries planned at lower levels. Update the expected plans that now use a Shared Scan, in cte_prune, cte_prune_multi_consumer, dedupset, gporca, shared_scan and table_functions, including queries in the ORCA answer files that fall back to the Postgres planner. Query results are unchanged.
chenjinbao1989
requested review from
avamingli,
lss602726449,
my-ship-it,
reshke and
yjhjstz
and removed request for
avamingli,
reshke and
yjhjstz
November 20, 2026 00:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A CTE referenced more than once is inlined by default: each reference is
planned as a separate subquery, with its own quals pushed down into it.
With gp_cte_sharing, all references share a single computation instead,
but then none of their quals can be pushed down. Which is better depends
on the query. At TPC-DS scale factor 10, with the Postgres planner,
gp_cte_sharing makes q23 1.8x faster, where both references are
"x IN (SELECT ... FROM cte)", but q04 5.9x, q11 6.0x and q31 6.2x slower.
In q31 each of three references to a CTE filters on a different quarter:
inlined, each copy pushes its filter down and the partition selectors skip
420 partition scans; shared, the CTE computes all quarters and years,
scanning 6.6 times as many store_sales rows. Over all 99 queries, turning
on gp_cte_sharing makes the total 15% slower.
Add gp_cte_sharing_auto (off by default; only used when gp_cte_sharing is
off). It decides per reference: a reference that has quals that can be
pushed down into the CTE is inlined, as without sharing, and the others
share a single computation of the CTE. The new
cte_ref_has_pushable_quals() makes the same checks as push_down_restrict(),
without pushing anything down, as push_down_restrict() removes the quals
it pushes from the rel's restrictions.
The CTE is shared only if its planned result is partitioned across the
segments. That is where sharing pays off, and otherwise a reference
executed in the QD and one executed in the segments may need the same
Shared Scan, which fails with "cannot share ShareInputScan between QD and
primary reader/write gang"; gp_cte_sharing fails like that, for example,
in gp_stat_reset_single_table_counters(). To check, the shared plan is
made before the decision; plan_shared_cte() is split out of
set_cte_pathlist() for that.
Like gp_cte_sharing, the option is turned off in subqueries planned at
lower levels, to avoid deadlocks between Shared Scans, and doesn't apply
to recursive CTEs, CTEs that aren't SELECTs, or CTEs with volatile
functions. MATERIALIZED and NOT MATERIALIZED still take precedence.
At scale factor 10, 8 of the 99 TPC-DS queries change plan, none gets
slower, and the total goes from 302.3 s to 275.4 s; q23 from 45.7 s to
25.3 s, q14 from 26.3 s to 22.5 s, q59 from 4.0 s to 3.0 s. The queries
that gp_cte_sharing makes slower don't change. The results are unchanged.
This is a heuristic. Where all references filter the CTE, they keep the
inlined plans, even if sharing would pay off anyway: gp_cte_sharing makes
q74 twice as fast, gp_cte_sharing_auto leaves it as it is.