Repository navigation
ci: back off longer when an infra image pull is rate limited - #1292
imranterranode wants to merge 1 commit into
Conversation
Infra / wab failed on three runs on 2026-10-21 because anonymous pulls of its base image from public.ecr.aws returned 429 Too Many Requests on all three build attempts, which retried after only 5 and 10 seconds. Its audit, build, lint and 202 tests had passed each time. When the build output shows a registry rate limit, retry up to five times after 15, 30, 60 and 120 seconds. Any other build failure keeps the existing three attempts after 5 and 10 seconds, so a real build error does not wait minutes to fail. A test runs the step with a stubbed docker and sleep to pin both paths.
|
|
@ty-everett quick question so we can fix this properly rather than keep retrying.
This PR is only a stopgap: it backs off longer so CI rides out the limit. Ideally we would not pull the base image anonymously on every run at all, either by:
Hence the question: is ECR deliberate for the AWS Marketplace build, or can |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |



Summary
This is a stopgap. It makes CI ride out AWS rate limits instead of failing. The goal is to stop
wabdepending on anonymouspublic.ecr.awspulls at all, by pointing it at the same Docker Hub base the other servers use (see the follow-up below, pending @ty-everett on whether ECR is needed for the AWS Marketplace build). Once that lands, this back-off should rarely if ever trigger.Infra / wabfailed on three CI runs on 2026-10-21 (e.g. run 39555124282 attempt 1, run 39560575480 attempts 1 and 2) because pulling its base image anonymously frompublic.ecr.awsreturned429 Too Many Requests. Its audit, build, lint and 202 tests had already passed each time. The "Build Linux image" step retried only three times, 5 and 10 seconds apart, so it never outlasted the rate limit, andmerge-gatefailed with it.What changes
.github/workflows/ci.yml, step "Build Linux image" in theinfrajob:429 Too Many Requestsortoomanyrequests), retry up to five times after 15, 30, 60 and 120 seconds.pipefailand tees the build output to$RUNNER_TEMPso the rate limit can be recognised.No image, Dockerfile, registry, scan or dependency changes.
Verification
scripts/ci-orchestration.test.mjsextracts the step script from the workflow text (no YAML dependency, as the Repository health job installs none) and runs it with a stubbeddockerandsleep:node --test scripts/*.test.mjs(364 pass),pnpm lint,pnpm format:root,node scripts/repository-health.mjs: pass.ci.ymlselects a full run, including every infra image build.Goal: remove the rate limit at its source (follow-up)
infra/wabis the only server whose Dockerfile pulls the Node base frompublic.ecr.aws; every other server uses the Docker Hub reference for the same digest (node@sha256:e67514e5…), and none of those failed. If ECR is not required for the AWS Marketplace build, switchingwabto the Docker Hub reference would remove the rate limit at its source. That is a supply-chain decision, so it is not part of this PR.Impact