Repository navigation
fix(security): build Hub with Go 1.27.3 and the patched CA store [ENG-4067] - #152
Conversation
Go 1.27.3 rejects chains through the compromised eMudhra intermediates in crypto/x509 (CVE-2026-111943), whatever roots are installed. The runtime image now also requires ca-certificates and ca-certificates-bundle 20261020-r0, which drop emSign ECC Root CA - G3. Both are pinned because plain `apk add ca-certificates` leaves the base image's older bundle package installed. Ref: ENG-4067 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WalkthroughThe Go version changes from 1.27.2 to 1.27.3 in the module declaration, CI workflows, linter configuration, and Docker builder image. The Dockerfile also adds minimum versions for the runtime ca-certificates and ca-certificates-bundle packages. Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to
|
There was a problem hiding this comment.
🔇 Additional comments: 9
go.mod: 1
3-3: LGTM!.github/workflows/api-contract-tests.yml: 1
56-56: LGTM!.github/workflows/code-quality.yml: 1
42-42: LGTM!.github/workflows/migrations-validate.yml: 1
28-28: LGTM!.github/workflows/tests.yml: 1
34-34: LGTM!Also applies to: 88-88, 151-151
.golangci.yml: 1
12-12: LGTM!Dockerfile: 2
5-5: LGTM!
58-59: 🔒 Security & PrivacyThe comment requests additional runtime validation but does not identify a demonstrated Dockerfile failure or a violated repository validation requirement. The reported absence of a final-image probe does not establish that the image fails to build or that the CA configuration is incorrect.
Dockerfile-58-59: 1
58-59: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.Pin the runtime packages consistently.
The command leaves
tzdataandwgetunpinned. It also uses>=for the CA packages, which permits later versions instead of selecting an exact version. If reproducible runtime images are required, use exact=constraints for all packages and remove the duplicate unversionedca-certificatesentry.Source: Linters/SAST tools
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Organization UI
- Review profile: ASSERTIVE
- Plan: Advanced
- Run ID:
81eee379-0c1c-4276-9ab2-e6b2aa0a0684
📒 Files selected for processing: 7
.github/workflows/api-contract-tests.yml.github/workflows/code-quality.yml.github/workflows/migrations-validate.yml.github/workflows/tests.yml.golangci.ymlDockerfilego.mod
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
alpine:3.24 still ships libssl3/libcrypto3 3.5.8, and 3.5.9 fixes High-severity CVEs. Only busybox wget's TLS and apk use OpenSSL in this image, but the pin costs nothing while the image is being rebuilt anyway. Ref: ENG-4067 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What does this PR do?
Rebuilds the Hub on Go 1.27.3 and the patched Alpine CA store, so the API and worker stop trusting the compromised eMudhra intermediates.
Ref ENG-4067
Context. The private keys of eight eMudhra intermediate CAs are recoverable (advisory). They chain to
emSign ECC Root CA - G3(viaemSign Root TLS CA - G3) and toAAA Certificate Services(via Sectigo cross-signs). Anyone could mint a certificate for any host on those chains, and our outbound HTTPS from the Hub would accept it. That covers the LLM providers, webhooks, taxonomy and OTLP.What changed
go.mod, the Dockerfile builder,.golangci.ymland everysetup-gostep. Go 1.27.3 makescrypto/x509drop any chain through those intermediates, whatever roots are installed (CVE-2026-111943, announcement). This is the actual fix. The Hub has no custom TLS config (noRootCAs,InsecureSkipVerifyorSSL_CERT_FILE), so every outbound client goes through this verifier.20261020-r0in the runtime image, which removesemSign ECC Root CA - G3. This is belt and braces for Go, and it also fixes busyboxwgetin the image. Bothca-certificatesandca-certificates-bundleare pinned on purpose: plainapk add ca-certificatesregenerates the bundle file but leaves the base image'sca-certificates-bundle-20260909-r0installed, soapk infowould report the old version (same pattern as the existing zlib pin).libssl3/libcrypto3) in the runtime image. This is unrelated to eMudhra: 3.5.9 fixes High-severity CVEs and alpine:3.24 doesn't ship it yet. Only busyboxwgetandapkuse OpenSSL here (the Go binaries don't), so it's a cheap pin while we rebuild anyway. With it,apk list -uin the final image comes back empty.hub-api,hub-worker,backfill-embeddings,gooseandriverall reportgo1.27.3.No API, schema or config changes.
Scope. This is the main line (the next 0.14.x release; formbricks main and 6.1 pin 0.14.0). The 0.8.x line (5.3, 5.4, 6.0 and KSA Cloud) needs Go 1.26.10. That patch is separate because its base (0.8.7 vs per-tag rebuilds) is still being decided. Rollout is tracked in ENG-4068.
How should this be tested?
What I ran locally (Docker, Go 1.27.3):
go vet ./..., unit tests (./cmd/api ./cmd/worker ./internal/...) and integration tests (./tests/...) against pgvector pg18, migrated with this image's own goose and river: all green. The pre-commit hook'sgolangci-lintalso passed with 0 issues.goose up+river migrate-up.ghcr.io/formbricks/hub:0.14.0(sha256:967089f2…, the same digest the chart pins and EU runs) and this build. I registered webhooks to the advisory's test hosts and created a feedback record. Results fromriver_job:dv-g3b-emudhra…(emSign route)x509verification errorev-g3c-emudhra…(emSign route)x509verification errordv-g3b-aaa-emudhra…(Sectigo cross-sign route)x509verification errorhttpbin.org/post(valid)self-signed.badssl.com{dv,ov,ev}-{g3b,g3c}[-aaa]-emudhra.we-have-a-problem.com), using a small Go probe run inside each image:busybox
wgetin the final image also rejects the compromised host and still reaches valid ones.In every run,
api.openai.comandgenerativelanguage.googleapis.comwere accepted and the self-signed and untrusted-root hosts were rejected.To reproduce:
Not verified here: production pods (ENG-4068 does that per region), and the TEI embeddings image, which is a separate Rust verifier outside this repo.
Checklist
Required
make build(built the Docker image, which compiles all shipped binaries)make tests(integration tests intests/)make fmtandmake lint; no new warningsgit pull origin mainmigrations/with goose annotations and ranmake migrate-validate(no schema change)Appreciated
docs/if changes were necessary (none needed)make tests-coveragefor meaningful logic changes (no logic change)