Repository navigation
Conversation
This optimization pushes down Bloom Filter conditions for runtime filters to the Pax Table AM layer. By applying the filter earlier than the SeqNext() function, it eliminates the overhead of converting data from columnar format to TableTupleSlot, resulting in faster query execution (cherry picked from commit e0abc24)
after calculating the null_counts array in advance, there is no need to call GetColumnDatum to continue updating null_counts. We can directly read the datum. (cherry picked from commit 6bc5d05)
Contributor
Author
|
Evaluating TPC-DS tests to show the overall effect - wait for the results to get ready for review |
There was a problem hiding this comment.
🟡 Changes recommended
Unmapped scan-key attributes can cause partition scans to discard valid join matches, and runtime filters published after a partition opens never reach its active child.
4 open issues
What changed in this PR
This PR extends runtime Bloom filtering with adaptive probing and rejection-rate ordering in sequential scans and PAX storage, aiming to reduce overhead from ineffective filters.
Changes:
- Adds sampling-based probe disabling and filter ranking.
- Integrates adaptive filtering into PAX row reads and moves dynamic-partition filtering into child scans.
- Adds regression and unit coverage for filtering, ordering, and partition scans.
| File | Description |
|---|---|
src/test/modules/test_bloomfilter/test_bloomfilter.c |
Tests probe disabling and sequential-scan ordering. |
src/test/modules/test_bloomfilter/sql/test_bloomfilter.sql |
Invokes the adaptive-probing test. |
src/test/modules/test_bloomfilter/expected/test_bloomfilter.out |
Updates expected test output. |
src/include/nodes/execnodes.h |
Adds sequential-scan sampling state. |
src/include/lib/bloomfilter.h |
Exposes adaptive probe APIs. |
src/include/executor/nodeSeqscan.h |
Updates the filtering function signature. |
src/backend/lib/bloomfilter.c |
Implements probe sampling, disabling, and ranking. |
src/backend/executor/nodeSeqscan.c |
Applies adaptive probes and orders filters. |
src/backend/executor/nodeHashjoin.c |
Uses relation attribute numbers for dynamic-scan filters. |
src/backend/executor/nodeDynamicSeqscan.c |
Attaches filters to child scans and aggregates statistics. |
contrib/pax_storage/src/test/regress/sql/gp_runtime_filter.sql |
Adds join and partition pushdown scenarios. |
contrib/pax_storage/src/test/regress/expected/gp_runtime_filter.out |
Records expected plans and filtering statistics. |
contrib/pax_storage/src/cpp/storage/orc/orc_group.cc |
Adjusts null-aware column value retrieval. |
contrib/pax_storage/src/cpp/storage/micro_partition_row_filter_reader.h |
Declares sampling and filter-evaluation helpers. |
contrib/pax_storage/src/cpp/storage/micro_partition_row_filter_reader.cc |
Evaluates Bloom and expression filters with sampling. |
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc |
Updates comments about filter handling. |
contrib/pax_storage/src/cpp/storage/filter/pax_row_filter.h |
Adds unified filter nodes and sampling state. |
contrib/pax_storage/src/cpp/storage/filter/pax_row_filter.cc |
Registers runtime filters and sorts sampled nodes. |
contrib/pax_storage/src/cpp/storage/filter/pax_filter.h |
Extends row-filter initialization parameters. |
contrib/pax_storage/src/cpp/storage/filter/pax_filter.cc |
Forwards scan keys to row-filter initialization. |
contrib/pax_storage/src/cpp/storage/filter/pax_filter_test.cc |
Tests sampled filter ordering. |
contrib/pax_storage/src/cpp/comm/cbdb_api.h |
Exposes Bloom APIs to PAX code. |
contrib/pax_storage/src/cpp/access/pax_scanner.cc |
Supplies scan keys when creating row filters. |
Review effort: Balanced
Sample non-null runtime Bloom probes and stop probing filters with low rejection rates. Rank sampled scan keys by their rejection rate and reuse PAX row-filter ordering and short-circuiting. Cover the cutoff and ordering in the backend and PAX tests.
leborchuk
force-pushed
the
adaptive-bloom-probing
branch
from
November 19, 2026 13:34
53e0e02 to
aa6b7c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


This is a extension of the original work #1333
With the adaptation to the anser #1942
Here I try to make bloom filters more effective. Use approach https://vldb.org/pvldb/vol19/p1278-qiao.pdf section 5
"Runtime Filter Skipping" to decide if I want to use specific bloom filter or not.
Also try to sort bloom filters by reject rate (first with the highest reject rate).
Use the same approach for PAX and NodeScan