Skip to content

Provider validation: lease fencing and exactly-once consumption #75

Open

Description

@waldemort-auto

Why

microsoft/duroxide-pg#44 fixes a provider bug that the validation suite did not catch, microsoft/duroxide-pg#29:

  1. An orchestration acknowledge committed after another dispatcher had already taken the instance over.
  2. The messages that acknowledge consumed stayed queued, and the next owner processed them again.
  3. Replay does not de-duplicate queue messages or external events. So, reproduced end to end with duroxide 0.1.32, one enqueued event ran an activity twice.

The suite already checks some related cases:

  • a stale token that is rejected before the acknowledge starts (lock_expiration::test_lock_expires_after_timeout, which acknowledges an empty turn);
  • two acknowledges with the same token (atomicity::test_concurrent_ack_prevention).

It does not check that ownership holds until the acknowledge commits. It also has no positive test for orchestration lock renewal; the worker side has three.

All tests below use only the Provider trait and ProviderFactory::lock_timeout() (written T below). They should pass on the SQLite provider.

Contract changes (src/providers/mod.rs, docs/provider-implementation-guide.md)

  1. ack_orchestration_item: the lock token must still own the instance when the acknowledge commits, not only when it starts. If ownership was lost at any point, the acknowledge fails with a permanent error and commits nothing: no history, no new work, no deleted messages.
  2. renew_orchestration_item_lock: once the token no longer owns the instance (after an acknowledge, an abandon or a takeover), renewal fails and changes nothing. It must not lock released messages again or change the new owner's lease.
  3. Decide whether renewal may shorten a lease. The documented pattern is locked_until = now() + extend_for. Proposed: locked_until = max(locked_until, now() + extend_for). duroxide-pg does this since fix: make orchestration lock renewal and acknowledge safe under concurrency duroxide-pg#44.

Tests to add

# Module Test Steps Must hold
1 lock_expiration test_ack_after_takeover_commits_nothing Fetch as A (lock T). Sleep past T. Fetch as B. Acknowledge as A with a non-empty history delta and one worker item A's acknowledge fails. read(instance) has none of A's events, and A's worker item was not enqueued. B's acknowledge succeeds, and no later fetch returns the message
2 lock_expiration test_orchestration_lock_renewal_extends_lock Fetch (lock T). Renew for 3T. Sleep 2T A second fetch returns nothing. An acknowledge with the original token succeeds
3 lock_expiration test_orchestration_renewal_after_takeover_fails Fetch as A. Wait for expiry. Fetch as B. Renew as A The renewal fails. A fetch right after it returns nothing. B's acknowledge succeeds and consumes the message once
4 lock_expiration test_orchestration_renewal_after_release_fails (a) Fetch, acknowledge, renew. (b) Fetch, abandon with no delay, renew Both renewals fail. In (b), a fetch right after the renewal returns the message: renewal did not lock it again
5 lock_expiration test_orchestration_renewal_never_shortens (after contract item 3) Fetch (lock T). Renew for 4T. Renew for T. Sleep 2T A second fetch returns nothing
6 instance_locking test_acknowledged_messages_never_redelivered_under_takeover_race N rounds (default 100). Each round: enqueue one QueueMessage. A fetches (lock T) and sleeps until about T, with jitter across rounds. Then A acknowledges a turn that consumes the message, while B polls fetch for up to 2T and acknowledges anything it gets with its own token. After every lock has expired, fetch once more Per round, at most one acknowledge that consumed the message succeeds. Once one succeeds, no later fetch returns the message

Notes:

Acceptance

Out of scope (these stay in duroxide-pg)

  • The clock-skew races, which need a faked caller clock that the trait cannot provide.
  • The PostgreSQL lock-order deadlock tests and the deadlock-victim retry.
  • The instance_locks(lock_token) index test.

These are in tests/deadlocks/ and tests/orchestration_*_tests.rs in microsoft/duroxide-pg#44. The TLA+ model in docs/tla/orchestration-leases there states the same properties formally.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions