Skip to content

ci: back off longer when an infra image pull is rate limited - #1292

Open
imranterranode wants to merge 1 commit into
mainfrom
ci/infra-image-pull-backoff
Open

imranterranode wants to merge 1 commit into
mainfrom
ci/infra-image-pull-backoff

Conversation

@imranterranode

@imranterranode imranterranode commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This is a stopgap. It makes CI ride out AWS rate limits instead of failing. The goal is to stop wab depending on anonymous public.ecr.aws pulls at all, by pointing it at the same Docker Hub base the other servers use (see the follow-up below, pending @ty-everett on whether ECR is needed for the AWS Marketplace build). Once that lands, this back-off should rarely if ever trigger.

Infra / wab failed on three CI runs on 2026-10-21 (e.g. run 39555124282 attempt 1, run 39560575480 attempts 1 and 2) because pulling its base image anonymously from public.ecr.aws returned 429 Too Many Requests. Its audit, build, lint and 202 tests had already passed each time. The "Build Linux image" step retried only three times, 5 and 10 seconds apart, so it never outlasted the rate limit, and merge-gate failed with it.

What changes

.github/workflows/ci.yml, step "Build Linux image" in the infra job:

  • When the build output shows a registry rate limit (429 Too Many Requests or toomanyrequests), retry up to five times after 15, 30, 60 and 120 seconds.
  • Any other build failure keeps the existing behaviour: three attempts after 5 and 10 seconds. A real build error still fails fast.
  • The step now runs with pipefail and tees the build output to $RUNNER_TEMP so the rate limit can be recognised.

No image, Dockerfile, registry, scan or dependency changes.

Verification

  • New test in scripts/ci-orchestration.test.mjs extracts the step script from the workflow text (no YAML dependency, as the Repository health job installs none) and runs it with a stubbed docker and sleep:
    • succeeds first time: 1 call, no waits
    • rate limited three times then succeeds: 4 calls, waits 15/30/60
    • rate limited every time: 5 calls, waits 15/30/60/120, fails
    • genuine build failure: 3 calls, waits 5/10, fails
  • node --test scripts/*.test.mjs (364 pass), pnpm lint, pnpm format:root, node scripts/repository-health.mjs: pass.
  • Hosted CI: pending on this PR. Changing ci.yml selects a full run, including every infra image build.

Goal: remove the rate limit at its source (follow-up)

infra/wab is the only server whose Dockerfile pulls the Node base from public.ecr.aws; every other server uses the Docker Hub reference for the same digest (node@sha256:e67514e5…), and none of those failed. If ECR is not required for the AWS Marketplace build, switching wab to the Docker Hub reference would remove the rate limit at its source. That is a supply-chain decision, so it is not part of this PR.

Impact

  • No public package source or manifest changed
  • Infrastructure source, dependency, image, or deployment configuration changed
  • No dependency or lockfile change
  • No new override, advisory dismissal, quality suppression, or skipped test
  • Workflow permissions and lifecycle-script behavior remain least privilege

Infra / wab failed on three runs on 2026-10-21 because anonymous pulls of
its base image from public.ecr.aws returned 429 Too Many Requests on all
three build attempts, which retried after only 5 and 10 seconds. Its
audit, build, lint and 202 tests had passed each time.

When the build output shows a registry rate limit, retry up to five
times after 15, 30, 60 and 120 seconds. Any other build failure keeps
the existing three attempts after 5 and 10 seconds, so a real build
error does not wait minutes to fail. A test runs the step with a stubbed
docker and sleep to pin both paths.
@sonarqubecloud

Copy link
Copy Markdown

@imranterranode

imranterranode commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

@ty-everett quick question so we can fix this properly rather than keep retrying.

infra/wab is the only server whose Dockerfile pulls the Node base from public.ecr.aws (same digest as the Docker Hub node@sha256:e67514e5… every other server uses). Anonymous ECR pulls from GitHub runners are getting 429 Too Many Requests; it failed Infra / wab on three CI runs today even though its tests passed.

This PR is only a stopgap: it backs off longer so CI rides out the limit. Ideally we would not pull the base image anonymously on every run at all, either by:

  1. Pointing wab at the Docker Hub reference like the other servers (same digest, so a byte-identical image), or
  2. Caching the base image in CI (it is digest-pinned, so a cached copy is guaranteed identical), or both.

Hence the question: is ECR deliberate for the AWS Marketplace build, or can wab switch to the Docker Hub reference? And are you happy for us to cache the base image in CI?

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant