Skip to content

pcgroups: Add consume error handler and recover from terminal consumer errors - #106

Open
arminazimi wants to merge 1 commit into
synadia-io:mainfrom
arminazimi:pcgroups-consume-err-handler
Open

arminazimi wants to merge 1 commit into
synadia-io:mainfrom
arminazimi:pcgroups-consume-err-handler

Conversation

@arminazimi

Copy link
Copy Markdown

Fixes #41

ElasticConsume and StaticConsume called Consume() without an error handler. When the member's consumer was terminated (e.g. deleted), nats.go stopped the consume context, but the member kept its place in the group without processing anything, and the caller had no way to find out.

Changes

  • New WithConsumeErrHandler(func(err error)) option, passed as a trailing variadic ...ConsumeOpt so existing callers keep compiling. It receives every error reported by the member's underlying consumer. A nil handler is rejected.
  • Terminal errors (ErrConsumerDeleted, ErrBadRequest, the ones after which nats.go stops the consume context) are now forwarded to the instance's control routine:
    • Elastic: the member drops its consumer, and the existing catch-all re-creates it and resumes consuming.
    • Static: the member stops and the error (wrapping the original one) is returned on Done().
  • Errors coming from a consume context that was already replaced (e.g. after a membership change) are ignored.

Behavior note

For the static group I chose to report through Done() rather than re-creating the durable consumer, since a deleted static consumer means it was removed from outside. Happy to change this if you prefer re-creating it.

Tests

New tests in pcgroups/test delete the member's consumer and check that the handler gets ErrConsumerDeleted, that the static member reports it on Done(), and that the elastic member resumes consuming. The elastic test times out without the fix. go test -race ./... passes in pcgroups/test.

…r errors

ElasticConsume and StaticConsume started the underlying Consume() without an
error handler, so when the member's consumer was terminated (e.g. deleted)
the member silently stopped processing messages while still holding its place
in the group, and the caller had no way to find out.

Add a WithConsumeErrHandler option (variadic, backwards compatible) that
receives the errors reported by the consumer. On a terminal error the
member's control routine now reacts: an elastic member drops its consumer so
the existing catch-all re-creates it and resumes consuming, a static member
stops and reports the error on Done().

Fixes synadia-io#41

Signed-off-by: a.azimi <arminazimi4030@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pcgroups: Zombie consumer on terminal error

1 participant