Skip to content

Add opt-in LISTEN/NOTIFY long polling - #41

Merged
Alperen Cetin (hacetin) merged 1 commit into
activity-admission-progressfrom
thomcc/long-polling
Nov 20, 2026
Merged

Alperen Cetin (hacetin) merged 1 commit into
activity-admission-progressfrom
thomcc/long-polling

Conversation

@thomcc-work

@thomcc-work Thom Chiovoloni (thomcc-work) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in PostgreSQL LISTEN/NOTIFY long polling for orchestration and activity work, adapted from duroxide-pg-opt. The goal is to reduce empty-queue queries and polling-induced latency while preserving existing queue eligibility, locking, and acknowledgment rules. Existing constructors continue to use short polling by default.

let mut config = ProviderConfig::url(database_url);
config.long_poll.enabled = true;
let provider = PostgresProvider::new_with_config(config).await?;

The same configuration is supported for Entra-authenticated connections. SQLx 0.8.3 or later is required for explicit listener reconnect handling; native-TLS configuration is unchanged.

The provider and its stress harness use the published Duroxide 0.2.0 dependency. Applications that depend directly on duroxide should use the 0.2 series with this provider.

Waiting and recovery

  • Register notification waiters before checking the queue, avoiding lost wakeups. Notifications wake all waiters for the affected queue so tag and session filters do not starve matching consumers.
  • Track delayed visibility and orchestration, activity, and session lease expiry. Future-dated lease hints allow recovery after an owner crashes without waiting for the full fetch poll timeout.
  • Notify after orchestration acknowledgment releases the instance lock. Delayed abandonment also sends an immediate release hint so other already-visible messages need not wait for the abandoned message’s delay.
  • Keep timer storage bounded and rescan after timers fire. Periodic recovery scans default to 60 seconds. The first availability wake is immediate; repeated unsuccessful wakes and timer catch-up scans are coalesced to a 100 ms cadence.
  • Run timer scans in READ ONLY transactions with 200 ms server-side lock and statement timeouts. Server-cancelled scans roll back and return their connections to the pool instead of accumulating blocked backends.
  • Keep at most one timer scan in flight, retaining it across watchdog wakeups and listener reconnects. A one-second polling fallback continues independently of stalled scans or reconnect attempts.
  • After resubscribing, recover deadlines with a fresh scan of both queues. If a scan from before the new LISTEN is still running, retain it and schedule the fresh catch-up scan after it completes.
  • Preserve zero-timeout fetch behavior. The poll deadline bounds notification waiting; it never cancels a potentially mutating database fetch.

Notifications remain hints: database predicates and locks determine whether work can actually be claimed.

Migration and activation

Landing order: #32, then #51, then #41. This top layer adds 0030 after the timestamp/pruning migrations (0027 and 0028) and the worker-progress migration (0029). Deploy the ordered sequence together; a ledger that skips a predecessor is rejected. Follow #32's fully drained upgrade when coming from before 0027.

The lower layers supply the Duroxide 0.2.0 bump, duplicate-acknowledgement validation, and deferred-worker ordering. This layer exercises the same validations with long polling enabled; it does not duplicate their registrations.

Migration 0030 installs six disabled notification triggers covering both queues and session leases. Notifications are activated schema-wide only by construction with both MigrationPolicy::ApplyAll and long polling enabled. VerifyOnly performs no DDL and requires prior activation. Disabling an individual provider’s listener does not disable shared triggers.

Migration and activation transactions use 250 ms lock timeouts and jittered retries for deadlocks and lock contention, with a ten-second retry budget. Failed attempts roll back without advancing the ledger. Migration errors retain the database message and SQLSTATE source. Connections are preserved for reuse only after successful migration and confirmed advisory-lock release; errors and cancellation close them.

The cleanup function preserves migration 0028’s batched-pruning cleanup. Disposable databases that ran an earlier draft of this PR as migration 0027 must be recreated. Older binaries still reject unknown migration versions even when notifications are disabled. This does not convert databases using the incompatible duroxide-pg-opt migration history.

Operational considerations

  • Long polling reserves one connection within the configured pool and requires at least two. It warms another connection for fetches. LISTEN requires a session-preserving connection; transaction-pooling proxies are not a substitute.
  • Activation adds notification costs to successful orchestration and activity claims, lease-changing renewals, and session-lease writes, not just enqueue and acknowledgment. Empty fetches do not notify. These costs also affect short-polling callers of an activated schema.
  • PostgreSQL serializes notification-emitting commits through a server-wide lock held through commit. WAL or synchronous-replication stalls can delay other notification writers, including in other databases. Evaluate representative write and renewal workloads before activation.
  • Transactions that emit notifications cannot use PREPARE TRANSACTION. Activated queue and session-lease writes are therefore incompatible with PostgreSQL two-phase commit.
  • Detected disconnects recover automatically, but silent network partitions are not actively probed. SQLx listener cleanup can still wait indefinitely on a wedged socket, so shutdown has no hard cleanup deadline in that case.
  • Providers must be recreated if their queue tables are dropped and recreated, because notification channels identify the original tables.

Validation

The provider suite runs with short and long polling. It now uses the released lock-expiration test from microsoft/duroxide#76: the checks start at the first successful claim, not at a task-start barrier. No lease duration or expiry margin was changed. The new duplicate-ack validation checks that retrying an already-committed activity acknowledgment fails permanently, with exactly one completion queued and the worker item removed.

With migrations 0027 through 0030 together, the deferred-worker validation exercises session and non-session work in both polling modes. The shared validation factory cleans up every schema it creates.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved wakeup gaps and schema-wide listener recovery require fixes and PostgreSQL-backed human validation.

2 open issues
What changed in this PR

Adds opt-in notification-driven long polling to the PostgreSQL provider while preserving short polling by default.

Changes:

  • Adds listener recovery, bounded timer tracking, and long-poll configuration.
  • Installs disabled notification triggers with explicit schema-wide activation.
  • Adds integration coverage, CI validation, and operational documentation.
File Description
tests/​postgres_provider_test.rs Enables validation in either polling mode.
tests/​long_poll_tests.rs Adds integration tests and an idle-query benchmark.
src/​provider.rs Integrates configuration and long-poll fetches.
src/​notifier.rs Implements listeners, timers, recovery, and waiting.
src/​migrations.rs Adds notification activation and verification.
src/​lib.rs Exports and documents long-poll configuration.
README.md Documents usage and operational limitations.
migrations/​README.md Documents activation and test commands.
migrations/​0027_diff.md Records migration changes.
migrations/​0027_add_long_poll_notifications.sql Installs disabled triggers and updates cleanup.
docs/​migration-policy.md Explains activation and compatibility.
CHANGELOG.md Records the unreleased feature.
Cargo.toml Raises the SQLx minimum to 0.8.3.
.github/​workflows/​test.yml Runs validation with long polling enabled.

Review effort: Balanced

Comment thread src/notifier.rs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Schema-wide trigger activation and concurrency-sensitive listener, timer, and cancellation behavior warrant human review backed by PostgreSQL integration results.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@thomcc-work
Thom Chiovoloni (thomcc-work) marked this pull request as ready for review October 26, 2026 16:46
Thom Chiovoloni (thomcc-work) added a commit that referenced this pull request Nov 2, 2026
Match #51's associated load_migrations and parse_version functions, including the failed-unlock regression added by #41. This keeps the standalone branch buildable and avoids E0599 when the prerequisite changes are combined.

Retain b33a3f2's connection-reuse behavior without changing pool sizing or tuning for the upstream barrier-based test. Addresses review item 41-N11.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@tjgreen42 Todd J. Green (tjgreen42) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by GPT-6.1 Sol.

No actionable findings in this revision. Reviewed listener startup/reconnection, wakeup registration and deferred-work deadlines, schema-wide trigger activation, polling fallback, shutdown, and migration compatibility. The earlier lost-wakeup/recovery concerns are addressed. The documented SQLx limitation around unbounded graceful pool.close() on a wedged listener is not a hang in provider/runtime drop, which aborts the listener without awaiting cleanup.

Scope: static review at 20dffa7, including the relevant SQLx cleanup behavior. No local PostgreSQL integration tests were run; local nextest compilation was blocked by the available Rust 1.88 toolchain versus a resolved dependency requiring Rust 1.89.

@tjgreen42

Copy link
Copy Markdown
Contributor

GPT-6.1 Sol: Please refresh the PR description to match the current branch. It still says the PR is a draft, but it is marked ready for review. Please also mention the later hardening: lease-expiry hints; read-only timer scans with 200 ms server-side timeouts; single-flight scans retained across watchdog wakeups and reconnects; and fresh catch-up scans of both queues after reconnect. The operational warning should explicitly include notification costs for successful claims, lease-changing renewals, and session-lease writes. The migration 0030 and prerequisite release-order notes still appear current.

@tjgreen42

Copy link
Copy Markdown
Contributor

Thom Chiovoloni (@thomcc-work) Are we planning to add the capability to migrate from a pre-existing duroxide-pg-opt installation?

@tjgreen42 Todd J. Green (tjgreen42) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, strictly an improvement on duroxide-pg-opt in several respects. Nice job. What's left before we can turn it on by default? (Stress-testing to make sure notifications don't get out of hand under high load?)

Short polling leaves workers idle between fetches. Use notifications and bounded visibility timers as wake-up hints, and keep the existing queue predicates and locks authoritative. If the listener or a timer scan stalls, polling can still recover ready work.

Install disabled triggers in migration 0030, after the pruning and worker-ordering migrations. Activate them only on explicit request. Run the existing provider validations against both polling modes without registering the runtime checks twice.

Source-Commit: 9c80e42

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hacetin
Alperen Cetin (hacetin) merged commit 6c4c756 into main Nov 20, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants