Skip to content

Adaptive bloom probing - #2359

Draft
leborchuk wants to merge 3 commits into
apache:mainfrom
leborchuk:adaptive-bloom-probing
Draft

leborchuk wants to merge 3 commits into
apache:mainfrom
leborchuk:adaptive-bloom-probing

Conversation

@leborchuk

Copy link
Copy Markdown
Contributor

This is a extension of the original work #1333

With the adaptation to the anser #1942

Here I try to make bloom filters more effective. Use approach https://vldb.org/pvldb/vol19/p1278-qiao.pdf section 5
"Runtime Filter Skipping" to decide if I want to use specific bloom filter or not.

Also try to sort bloom filters by reject rate (first with the highest reject rate).

Use the same approach for PAX and NodeScan

This optimization pushes down Bloom Filter conditions for runtime filters
to the Pax Table AM layer.

By applying the filter earlier than the SeqNext() function, it eliminates
the overhead of converting data from columnar format to TableTupleSlot,
resulting in faster query execution

(cherry picked from commit e0abc24)
after calculating the null_counts array in advance, there is no need to call
GetColumnDatum to continue updating null_counts. We can directly read the datum.

(cherry picked from commit 6bc5d05)
Copilot AI balanced review requested due to automatic review settings November 19, 2026 10:00
@leborchuk
leborchuk marked this pull request as draft November 19, 2026 10:00
@leborchuk

Copy link
Copy Markdown
Contributor Author

Evaluating TPC-DS tests to show the overall effect - wait for the results to get ready for review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unmapped scan-key attributes can cause partition scans to discard valid join matches, and runtime filters published after a partition opens never reach its active child.

4 open issues
What changed in this PR

This PR extends runtime Bloom filtering with adaptive probing and rejection-rate ordering in sequential scans and PAX storage, aiming to reduce overhead from ineffective filters.

Changes:

  • Adds sampling-based probe disabling and filter ranking.
  • Integrates adaptive filtering into PAX row reads and moves dynamic-partition filtering into child scans.
  • Adds regression and unit coverage for filtering, ordering, and partition scans.
File Description
src/​test/​modules/​test_bloomfilter/​test_bloomfilter.c Tests probe disabling and sequential-scan ordering.
src/​test/​modules/​test_bloomfilter/​sql/​test_bloomfilter.sql Invokes the adaptive-probing test.
src/​test/​modules/​test_bloomfilter/​expected/​test_bloomfilter.out Updates expected test output.
src/​include/​nodes/​execnodes.h Adds sequential-scan sampling state.
src/​include/​lib/​bloomfilter.h Exposes adaptive probe APIs.
src/​include/​executor/​nodeSeqscan.h Updates the filtering function signature.
src/​backend/​lib/​bloomfilter.c Implements probe sampling, disabling, and ranking.
src/​backend/​executor/​nodeSeqscan.c Applies adaptive probes and orders filters.
src/​backend/​executor/​nodeHashjoin.c Uses relation attribute numbers for dynamic-scan filters.
src/​backend/​executor/​nodeDynamicSeqscan.c Attaches filters to child scans and aggregates statistics.
contrib/​pax_storage/​src/​test/​regress/​sql/​gp_runtime_filter.sql Adds join and partition pushdown scenarios.
contrib/​pax_storage/​src/​test/​regress/​expected/​gp_runtime_filter.out Records expected plans and filtering statistics.
contrib/​pax_storage/​src/​cpp/​storage/​orc/​orc_group.cc Adjusts null-aware column value retrieval.
contrib/​pax_storage/​src/​cpp/​storage/​micro_partition_row_filter_reader.h Declares sampling and filter-evaluation helpers.
contrib/​pax_storage/​src/​cpp/​storage/​micro_partition_row_filter_reader.cc Evaluates Bloom and expression filters with sampling.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_sparse_pg_path.cc Updates comments about filter handling.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_row_filter.h Adds unified filter nodes and sampling state.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_row_filter.cc Registers runtime filters and sorts sampled nodes.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_filter.h Extends row-filter initialization parameters.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_filter.cc Forwards scan keys to row-filter initialization.
contrib/​pax_storage/​src/​cpp/​storage/​filter/​pax_filter_test.cc Tests sampled filter ordering.
contrib/​pax_storage/​src/​cpp/​comm/​cbdb_api.h Exposes Bloom APIs to PAX code.
contrib/​pax_storage/​src/​cpp/​access/​pax_scanner.cc Supplies scan keys when creating row filters.

Review effort: Balanced

Sample non-null runtime Bloom probes and stop probing filters with
low rejection rates. Rank sampled scan keys by their rejection rate
and reuse PAX row-filter ordering and short-circuiting. Cover the
cutoff and ordering in the backend and PAX tests.
@leborchuk
leborchuk force-pushed the adaptive-bloom-probing branch from 53e0e02 to aa6b7c6 Compare November 19, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants