Repository navigation
Open
Description
Why
microsoft/duroxide-pg#44 fixes a provider bug that the validation suite did not catch, microsoft/duroxide-pg#29:
- An orchestration acknowledge committed after another dispatcher had already taken the instance over.
- The messages that acknowledge consumed stayed queued, and the next owner processed them again.
- Replay does not de-duplicate queue messages or external events. So, reproduced end to end with duroxide 0.1.32, one enqueued event ran an activity twice.
The suite already checks some related cases:
- a stale token that is rejected before the acknowledge starts (
lock_expiration::test_lock_expires_after_timeout, which acknowledges an empty turn); - two acknowledges with the same token (
atomicity::test_concurrent_ack_prevention).
It does not check that ownership holds until the acknowledge commits. It also has no positive test for orchestration lock renewal; the worker side has three.
All tests below use only the Provider trait and ProviderFactory::lock_timeout() (written T below). They should pass on the SQLite provider.
Contract changes (src/providers/mod.rs, docs/provider-implementation-guide.md)
ack_orchestration_item: the lock token must still own the instance when the acknowledge commits, not only when it starts. If ownership was lost at any point, the acknowledge fails with a permanent error and commits nothing: no history, no new work, no deleted messages.renew_orchestration_item_lock: once the token no longer owns the instance (after an acknowledge, an abandon or a takeover), renewal fails and changes nothing. It must not lock released messages again or change the new owner's lease.- Decide whether renewal may shorten a lease. The documented pattern is
locked_until = now() + extend_for. Proposed:locked_until = max(locked_until, now() + extend_for). duroxide-pg does this since fix: make orchestration lock renewal and acknowledge safe under concurrency duroxide-pg#44.
Tests to add
| # | Module | Test | Steps | Must hold |
|---|---|---|---|---|
| 1 | lock_expiration |
test_ack_after_takeover_commits_nothing |
Fetch as A (lock T). Sleep past T. Fetch as B. Acknowledge as A with a non-empty history delta and one worker item | A's acknowledge fails. read(instance) has none of A's events, and A's worker item was not enqueued. B's acknowledge succeeds, and no later fetch returns the message |
| 2 | lock_expiration |
test_orchestration_lock_renewal_extends_lock |
Fetch (lock T). Renew for 3T. Sleep 2T | A second fetch returns nothing. An acknowledge with the original token succeeds |
| 3 | lock_expiration |
test_orchestration_renewal_after_takeover_fails |
Fetch as A. Wait for expiry. Fetch as B. Renew as A | The renewal fails. A fetch right after it returns nothing. B's acknowledge succeeds and consumes the message once |
| 4 | lock_expiration |
test_orchestration_renewal_after_release_fails |
(a) Fetch, acknowledge, renew. (b) Fetch, abandon with no delay, renew | Both renewals fail. In (b), a fetch right after the renewal returns the message: renewal did not lock it again |
| 5 | lock_expiration |
test_orchestration_renewal_never_shortens (after contract item 3) |
Fetch (lock T). Renew for 4T. Renew for T. Sleep 2T | A second fetch returns nothing |
| 6 | instance_locking |
test_acknowledged_messages_never_redelivered_under_takeover_race |
N rounds (default 100). Each round: enqueue one QueueMessage. A fetches (lock T) and sleeps until about T, with jitter across rounds. Then A acknowledges a turn that consumes the message, while B polls fetch for up to 2T and acknowledges anything it gets with its own token. After every lock has expired, fetch once more |
Per round, at most one acknowledge that consumed the message succeeds. Once one succeeds, no later fetch returns the message |
Notes:
- Tests 1 to 4 document the contract. A provider that checks the token only at the start of an acknowledge still passes them, because the takeover happens before the call.
- Test 5 fails on duroxide-pg
main(3beba92), whose renewal can shorten a lease. - Test 6 is the one that targets ack_orchestration_item validates the lock without a row lock; a concurrent fetch can leave consumed messages in the queue duroxide-pg#29. It is timing-based: measure its failure rate on duroxide-pg
3beba92before relying on it. If the window is too narrow to hit reliably, keep it as a soak test with a configurable round count.
Acceptance
- Tests 1 to 4, and test 5 once the contract is decided, pass on SQLite and on duroxide-pg with fix: make orchestration lock renewal and acknowledge safe under concurrency duroxide-pg#44.
- Test 5 fails on duroxide-pg
3beba92. - Test 6 passes on fix: make orchestration lock renewal and acknowledge safe under concurrency duroxide-pg#44, and its failure rate on
3beba92is recorded in the PR. - After the next duroxide release, duroxide-pg registers the new tests in
tests/postgres_provider_test.rs.
Out of scope (these stay in duroxide-pg)
- The clock-skew races, which need a faked caller clock that the trait cannot provide.
- The PostgreSQL lock-order deadlock tests and the deadlock-victim retry.
- The
instance_locks(lock_token)index test.
These are in tests/deadlocks/ and tests/orchestration_*_tests.rs in microsoft/duroxide-pg#44. The TLA+ model in docs/tla/orchestration-leases there states the same properties formally.
Related
- Tracking: lock loss, replay and commit edge cases (core, duroxide-pg and the SDKs) #55: tracking issue.
- A failed commit becomes a permanent OrchestrationFailed without parent notification; the fallback abandon never runs #58: the fallback abandon is dropped without
.await. - A turn that lost its orchestration lock keeps running, and the same runtime can start a second replay of the instance #59: a turn keeps running after it loses its lock.
- ack_orchestration_item validates the lock without a row lock; a concurrent fetch can leave consumed messages in the queue duroxide-pg#29 and fix: make orchestration lock renewal and acknowledge safe under concurrency duroxide-pg#44.
Metadata
Metadata
Assignees
Labels
No labels
Activity