Skip to content

Optionally share CTE references that have no quals to push down - #2361

Open
chenjinbao1989 wants to merge 2 commits into
apache:mainfrom
chenjinbao1989:pr-auto-cte-sharing
Open

chenjinbao1989 wants to merge 2 commits into
apache:mainfrom
chenjinbao1989:pr-auto-cte-sharing

Conversation

@chenjinbao1989

Copy link
Copy Markdown
Contributor

A CTE referenced more than once is inlined by default: each reference is
planned as a separate subquery, with its own quals pushed down into it.
With gp_cte_sharing, all references share a single computation instead,
but then none of their quals can be pushed down. Which is better depends
on the query. At TPC-DS scale factor 10, with the Postgres planner,
gp_cte_sharing makes q23 1.8x faster, where both references are
"x IN (SELECT ... FROM cte)", but q04 5.9x, q11 6.0x and q31 6.2x slower.
In q31 each of three references to a CTE filters on a different quarter:
inlined, each copy pushes its filter down and the partition selectors skip
420 partition scans; shared, the CTE computes all quarters and years,
scanning 6.6 times as many store_sales rows. Over all 99 queries, turning
on gp_cte_sharing makes the total 15% slower.

Add gp_cte_sharing_auto (off by default; only used when gp_cte_sharing is
off). It decides per reference: a reference that has quals that can be
pushed down into the CTE is inlined, as without sharing, and the others
share a single computation of the CTE. The new
cte_ref_has_pushable_quals() makes the same checks as push_down_restrict(),
without pushing anything down, as push_down_restrict() removes the quals
it pushes from the rel's restrictions.

The CTE is shared only if its planned result is partitioned across the
segments. That is where sharing pays off, and otherwise a reference
executed in the QD and one executed in the segments may need the same
Shared Scan, which fails with "cannot share ShareInputScan between QD and
primary reader/write gang"; gp_cte_sharing fails like that, for example,
in gp_stat_reset_single_table_counters(). To check, the shared plan is
made before the decision; plan_shared_cte() is split out of
set_cte_pathlist() for that.

Like gp_cte_sharing, the option is turned off in subqueries planned at
lower levels, to avoid deadlocks between Shared Scans, and doesn't apply
to recursive CTEs, CTEs that aren't SELECTs, or CTEs with volatile
functions. MATERIALIZED and NOT MATERIALIZED still take precedence.

At scale factor 10, 8 of the 99 TPC-DS queries change plan, none gets
slower, and the total goes from 302.3 s to 275.4 s; q23 from 45.7 s to
25.3 s, q14 from 26.3 s to 22.5 s, q59 from 4.0 s to 3.0 s. The queries
that gp_cte_sharing makes slower don't change. The results are unchanged.

This is a heuristic. Where all references filter the CTE, they keep the
inlined plans, even if sharing would pay off anyway: gp_cte_sharing makes
q74 twice as fast, gp_cte_sharing_auto leaves it as it is.

A CTE referenced more than once is inlined by default: each reference is
planned as a separate subquery, with its own quals pushed down into it.
With gp_cte_sharing, all references share a single computation instead,
but then none of their quals can be pushed down. Which is better depends
on the query. At TPC-DS scale factor 10, with the Postgres planner,
gp_cte_sharing makes q23 1.8x faster, where both references are
"x IN (SELECT ... FROM cte)", but q04 5.9x, q11 6.0x and q31 6.2x slower.
In q31 each of three references to a CTE filters on a different quarter:
inlined, each copy pushes its filter down and the partition selectors skip
420 partition scans; shared, the CTE computes all quarters and years,
scanning 6.6 times as many store_sales rows. Over all 99 queries, turning
on gp_cte_sharing makes the total 15% slower.

Add gp_cte_sharing_auto (off by default; only used when gp_cte_sharing is
off). It decides per reference: a reference that has quals that can be
pushed down into the CTE is inlined, as without sharing, and the others
share a single computation of the CTE. The new
cte_ref_has_pushable_quals() makes the same checks as push_down_restrict(),
without pushing anything down, as push_down_restrict() removes the quals
it pushes from the rel's restrictions.

The CTE is shared only if its planned result is partitioned across the
segments. That is where sharing pays off, and otherwise a reference
executed in the QD and one executed in the segments may need the same
Shared Scan, which fails with "cannot share ShareInputScan between QD and
primary reader/write gang"; gp_cte_sharing fails like that, for example,
in gp_stat_reset_single_table_counters(). To check, the shared plan is
made before the decision; plan_shared_cte() is split out of
set_cte_pathlist() for that.

Like gp_cte_sharing, the option is turned off in subqueries planned at
lower levels, to avoid deadlocks between Shared Scans, and doesn't apply
to recursive CTEs, CTEs that aren't SELECTs, or CTEs with volatile
functions. MATERIALIZED and NOT MATERIALIZED still take precedence.

At scale factor 10, 8 of the 99 TPC-DS queries change plan, none gets
slower, and the total goes from 302.3 s to 275.4 s; q23 from 45.7 s to
25.3 s, q14 from 26.3 s to 22.5 s, q59 from 4.0 s to 3.0 s. The queries
that gp_cte_sharing makes slower don't change. The results are unchanged.

This is a heuristic. Where all references filter the CTE, they keep the
inlined plans, even if sharing would pay off anyway: gp_cte_sharing makes
q74 twice as fast, gp_cte_sharing_auto leaves it as it is.
Drop gp_cte_sharing_auto: when gp_cte_sharing is off, the planner now
always decides per reference whether to share a CTE. A reference with
quals that can be pushed down into the CTE is still inlined, so the
queries that gp_cte_sharing makes slower keep their plans.

The per-level switch in PlannerConfig is renamed cte_sharing_auto; it
is still turned off in subqueries planned at lower levels.

Update the expected plans that now use a Shared Scan, in cte_prune,
cte_prune_multi_consumer, dedupset, gporca, shared_scan and
table_functions, including queries in the ORCA answer files that fall
back to the Postgres planner. Query results are unchanged.
@chenjinbao1989
chenjinbao1989 requested review from avamingli, lss602726449, my-ship-it, reshke and yjhjstz and removed request for avamingli, reshke and yjhjstz November 20, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant