Repository navigation
Prevent deferred worker items from starving ready work - #51
Alperen Cetin (hacetin) merged 1 commit into
Conversation
aca6832 to
9ae067b
Compare
Keep the database message in ordinary migration error display while retaining the SQLx source for SQLSTATE retry classification. Add a red/green regression that also proves failed migration statements leave the ledger, stored function, and durable rows unchanged. State that migration 0030 must not ship before the 0027-0029 prerequisites, including the dependency of #51 on #32. Addresses review items 41-N6 and 41-N7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match #51's associated load_migrations and parse_version functions, including the failed-unlock regression added by #41. This keeps the standalone branch buildable and avoids E0599 when the prerequisite changes are combined. Retain b33a3f2's connection-reuse behavior without changing pool sizing or tuning for the upstream barrier-based test. Addresses review item 41-N11. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pinned to 9ae067b, alongside the merged duroxide revision. Changes: - Updated the provider dependency and lockfile. - Advanced WORKER_SCHEMA_VERSION to 4 for the bundled migrations. - Added a 500-request backlog regression. - Updated migration, rollout, and behavior documentation.
Todd J. Green (tjgreen42)
left a comment
There was a problem hiding this comment.
Review by GPT-6.1 Sol.
No actionable correctness findings in this revision. Reviewed the visibility-first dequeue ordering, claim/session eligibility and locking, migration registration and validation, and starvation regression coverage.
Release coordination note: do not ship/apply migration 0029 before #32 supplies 0027/0028; a later combined build would reject that applied ledger as a non-prefix. This dependency is already documented in #41, but would also be useful in this PR description, which still calls the migration 0027.
Scope: static review at 9ae067b. No PostgreSQL integration tests were run. Local nextest validation was blocked by the resolved uuid dependency requiring Rust 1.89 while the available toolchain was 1.88.
|
Review by GPT-6.1 Sol: The PR description is stale relative to the current diff. Please update it to:
The core ordering behavior and provider compatibility summary remain accurate. |
The released lock-expiration validation measures from the first claim, so startup scheduling no longer consumes its expiry margin. Register the duplicate activity acknowledgment check as well: retrying after a lost commit response must not enqueue a second completion. Deferred-worker ordering depends on migration 0029 from #51. Validate it with the prerequisite migrations rather than duplicating their SQL in the long-poll branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The runtime now retries activity completion acknowledgements, including after a committed response was lost. Register the upstream duplicate-ack validation to ensure those retries cannot enqueue a second completion, and align the heterogeneous-worker test with at-least-once delivery. Keep the worker-queue progress implementation in the separately coordinated PR #51; retain this PR's migrations 0027/0028 and the current package release metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A deferred item keeps its queue ID, so ordering by ID lets it reclaim a worker before older ready work each time its delay expires. Order both worker fetch paths by visibility time, then ID. Validate the matching index before recording migration 0029 so a failed concurrent prebuild cannot silently change fetch performance. Build on the 0.2.0 runtime and duplicate-acknowledgement check from the timestamp and pruning changes. The deferred-work validation creates providers with and without sessions; retain every schema for cleanup. Source-Commit: d7a6369 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
d7a6369 to
142584d
Compare
* Record `completed_at` for `ContinuedAsNew` executions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add bounded cursor-based pruning, and protect pending continuations from age-filtered bulk deletion Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback * Fix lock order issue * Fix lock acquisition * Avoid locking protected executions during batch pruning Precheck eligibility without changing scanned-key bounds or cursors, then retain the locked eligibility recheck. Exercise the hostile search_path with an eligible row and verify protected instances do not require locks. Ref: #32 (32-S1, 32-S3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Report malformed history keys during continue-as-new backfill Keep the backfill set-based on success and preserve SQLSTATE while adding the failing history key on JSONB decode errors. Verify atomic rollback, valid upgrades, and a schema built entirely from the pre-0027 migration prefix. Ref: #32 (backfill diagnostics and upgrade-fixture review notes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Document fully drained upgrades and timestamp repair Require draining all provider callers, pause retention during upgrades, and describe the one-time deletion wave and backfill/WAL cost. Document an explicit timestamp-only repair after draining, preserve the migration ledger, and classify existing-procedure fixes under Fixed. Ref: #32 (32-M1, 32-S2, changelog review note) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify which process applies migrations during upgrades Start one new ApplyAll process first and wait for provider initialization to complete before restarting the remaining upgraded callers. Keep migrations numbered 0027/0028 pending merge-order coordination. Ref: #32 (upgrade-startup review nit) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use released Duroxide 0.2.0 in the provider and stress harness The runtime now retries activity completion acknowledgements, including after a committed response was lost. Register the upstream duplicate-ack validation to ensure those retries cannot enqueue a second completion, and align the heterogeneous-worker test with at-least-once delivery. Keep the worker-queue progress implementation in the separately coordinated PR #51; retain this PR's migrations 0027/0028 and the current package release metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Duroxide 0.2.0 lets an activity defer execution until a resource is available without holding a worker slot (microsoft/duroxide#74). PostgreSQL previously fetched eligible work by queue ID alone. A deferred item kept its old ID, so if its delay expired before the backlog drained, it could repeatedly jump ahead of work that was already ready.
Migration
0029orders eligible items byvisible_at, thenid, for both session and non-session fetches. Deferring a row moves it behind already-ready work; queue ID breaks ties. Session ownership, tag filtering, lock handling, and attempt accounting are unchanged. The companion migration diff comparesfetch_work_itemwith its0016baseline and shows the new index.Landing order: #32, then #51, then #41. #32 supplies Duroxide 0.2.0 for the provider and stress harness. This PR adds PostgreSQL queue ordering for the runtime's activity-admission API.
Migration and rollout
0029keeps thefetch_work_itemsignature and return shape and addsidx_worker_ready_orderonworker_queue(visible_at, id). On a busy queue, prebuild the matching index withCREATE INDEX CONCURRENTLYin each target schema, outside a transaction and before running the migration. A failed concurrent build can leave an invalid index; the migration rejects it without changing the migration ledger, fetch function, or queued work. See the rollout notes for the commands to prebuild, check, and repair the index.Prebuilding skips the index build, not the lock. Even when it reuses the index, the migration takes a
SHARElock onworker_queueuntil transaction commit. It blocks worker fetch, acknowledge, abandon, and lock renewal, as well as orchestration acknowledgements that write worker rows. The migration sets nolock_timeout; configure one on the deployment session to bound the wait for the lock.If only
0029is pending, the function remains compatible with already-running providers; older binaries still refuse to restart against the new ledger. Upgrading from before0027also requires the fully drained procedure in #32. Bundle migrations in order; do not edit the ledger to skip prerequisites.Validation
This layer registers the deferred-worker validation (an expired deferral yields to ready work and restores its fetch attempt) and cleans up both schemas it uses. The duplicate-acknowledgement validation is inherited from Record
completed_atforContinuedAsNewexecutions #32.PostgreSQL-specific regressions cover tagged and untagged work, invalid-index rejection with unchanged state, reuse of a valid concurrent prebuild, and unique, strictly increasing embedded migration versions.
At
d7a6369on PostgreSQL 17: 403 workspace tests passed, 9 skipped; all 8 doctests passed. The two upstream validations and both worker-progress regressions also passed ten consecutive runs without leaving test schemas behind.GitHub CI at
d7a6369: Cargo tests passed. The live Entra job stopped at Azure login before running the test: the CI application is disabled (AADSTS7000112).