Skip to content

fix(security): build Hub with Go 1.27.3 and the patched CA store [ENG-4067] - #152

Merged
xernobyl merged 2 commits into
mainfrom
fix/ENG-4067_go-1.27.3-ca-certificates
Oct 21, 2026
Merged

xernobyl merged 2 commits into
mainfrom
fix/ENG-4067_go-1.27.3-ca-certificates

Conversation

@xernobyl

@xernobyl xernobyl commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Rebuilds the Hub on Go 1.27.3 and the patched Alpine CA store, so the API and worker stop trusting the compromised eMudhra intermediates.

Ref ENG-4067

Context. The private keys of eight eMudhra intermediate CAs are recoverable (advisory). They chain to emSign ECC Root CA - G3 (via emSign Root TLS CA - G3) and to AAA Certificate Services (via Sectigo cross-signs). Anyone could mint a certificate for any host on those chains, and our outbound HTTPS from the Hub would accept it. That covers the LLM providers, webhooks, taxonomy and OTLP.

What changed

  • Go 1.27.2 → 1.27.3 in go.mod, the Dockerfile builder, .golangci.yml and every setup-go step. Go 1.27.3 makes crypto/x509 drop any chain through those intermediates, whatever roots are installed (CVE-2026-111943, announcement). This is the actual fix. The Hub has no custom TLS config (no RootCAs, InsecureSkipVerify or SSL_CERT_FILE), so every outbound client goes through this verifier.
  • CA store pinned to 20261020-r0 in the runtime image, which removes emSign ECC Root CA - G3. This is belt and braces for Go, and it also fixes busybox wget in the image. Both ca-certificates and ca-certificates-bundle are pinned on purpose: plain apk add ca-certificates regenerates the bundle file but leaves the base image's ca-certificates-bundle-20260909-r0 installed, so apk info would report the old version (same pattern as the existing zlib pin).
  • OpenSSL 3.5.8 → 3.5.9 (libssl3/libcrypto3) in the runtime image. This is unrelated to eMudhra: 3.5.9 fixes High-severity CVEs and alpine:3.24 doesn't ship it yet. Only busybox wget and apk use OpenSSL here (the Go binaries don't), so it's a cheap pin while we rebuild anyway. With it, apk list -u in the final image comes back empty.
  • Rebuilding covers everything shipped in the image: hub-api, hub-worker, backfill-embeddings, goose and river all report go1.27.3.

No API, schema or config changes.

Scope. This is the main line (the next 0.14.x release; formbricks main and 6.1 pin 0.14.0). The 0.8.x line (5.3, 5.4, 6.0 and KSA Cloud) needs Go 1.26.10. That patch is separate because its base (0.8.7 vs per-tag rebuilds) is still being decided. Rollout is tracked in ENG-4068.

How should this be tested?

What I ran locally (Docker, Go 1.27.3):

  • go vet ./..., unit tests (./cmd/api ./cmd/worker ./internal/...) and integration tests (./tests/...) against pgvector pg18, migrated with this image's own goose and river: all green. The pre-commit hook's golangci-lint also passed with 0 issues.
  • Migrations 001–027 apply cleanly with the candidate image's goose up + river migrate-up.
  • The shipped binaries' own TLS path. I ran the API and worker from both the published ghcr.io/formbricks/hub:0.14.0 (sha256:967089f2…, the same digest the chart pins and EU runs) and this build. I registered webhooks to the advisory's test hosts and created a feedback record. Results from river_job:
Webhook target 0.14.0 (published) This PR
dv-g3b-emudhra… (emSign route) ⚠️ TLS accepted, host replied 405 ✅ x509 verification error
ev-g3c-emudhra… (emSign route) ⚠️ TLS accepted, host replied 405 ✅ x509 verification error
dv-g3b-aaa-emudhra… (Sectigo cross-sign route) rejected (AAA isn't in Alpine's store) ✅ x509 verification error
httpbin.org/post (valid) delivered ✅ delivered
self-signed.badssl.com rejected ✅ rejected
  • All 12 published chains ({dv,ov,ev}-{g3b,g3c}[-aaa]-emudhra.we-have-a-problem.com), using a small Go probe run inside each image:
Image Probe built with Go 1.27.3 Probe built with Go 1.27.2 (isolates the CA store)
0.14.0 0/12 accepted 6/12 accepted
This PR 0/12 accepted 0/12 accepted

busybox wget in the final image also rejects the compromised host and still reaches valid ones.

In every run, api.openai.com and generativelanguage.googleapis.com were accepted and the self-signed and untrusted-root hosts were rejected.

To reproduce:

docker build -t hub-eng4067 .
docker run --rm --entrypoint sh hub-eng4067 -c 'apk info -v | grep -E "^(ca-cert|libssl3|libcrypto3)"'   # CA 20261020-r0, OpenSSL 3.5.9-r0
curl -sS https://dv-g3b-emudhra.we-have-a-problem.com/   # any client: a certificate error means it is protected

Not verified here: production pods (ENG-4068 does that per region), and the TEI embeddings image, which is a separate Rust verifier outside this repo.

Checklist

Required

  • Filled out the "How to test" section in this PR
  • Read Repository Guidelines
  • Self-reviewed my own code
  • Commented on my code in hard-to-understand bits
  • Ran make build (built the Docker image, which compiles all shipped binaries)
  • Ran make tests (integration tests in tests/)
  • Ran make fmt and make lint; no new warnings
  • Removed debug prints / temporary logging
  • Merged the latest changes from main onto my branch with git pull origin main
  • If database schema changed: added migration in migrations/ with goose annotations and ran make migrate-validate (no schema change)

Appreciated

  • If API changed: added or updated OpenAPI spec and ran contract tests (no API change)
  • If API behavior changed: added request/response examples or Swagger UI screenshots to this PR (no API change)
  • Updated docs in docs/ if changes were necessary (none needed)
  • Ran make tests-coverage for meaningful logic changes (no logic change)

Go 1.27.3 rejects chains through the compromised eMudhra intermediates
in crypto/x509 (CVE-2026-111943), whatever roots are installed. The
runtime image now also requires ca-certificates and
ca-certificates-bundle 20261020-r0, which drop emSign ECC Root CA - G3.
Both are pinned because plain `apk add ca-certificates` leaves the base
image's older bundle package installed.

Ref: ENG-4067

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xernobyl
xernobyl marked this pull request as ready for review October 21, 2026 08:55
@xernobyl
xernobyl marked this pull request as draft October 21, 2026 08:56
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Walkthrough

The Go version changes from 1.27.2 to 1.27.3 in the module declaration, CI workflows, linter configuration, and Docker builder image. The Dockerfile also adds minimum versions for the runtime ca-certificates and ca-certificates-bundle packages.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 27db6

The toolchain and CA package updates have no established merge-blocking risk.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly identifies the security fix, the Go version upgrade, and the patched CA store. It follows the repository's Conventional Commits format.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The pull request description is complete and follows the repository template. It explains the security objective, lists the affected files and dependencies, references ENG-4067, documents testing and …
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (7 skipped: 7 unsupported.)

Full details: Description check

Explanation

The pull request description is complete and follows the repository template. It explains the security objective, lists the affected files and dependencies, references ENG-4067, documents testing and reproduction steps, and records checklist status. It also clearly notes that no API, schema, or configuration changes were made. The pull request title cannot be assessed because it was not provided.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔇 Additional comments: 9
go.mod: 1

3-3: LGTM!

.github/workflows/api-contract-tests.yml: 1

56-56: LGTM!

.github/workflows/code-quality.yml: 1

42-42: LGTM!

.github/workflows/migrations-validate.yml: 1

28-28: LGTM!

.github/workflows/tests.yml: 1

34-34: LGTM!

Also applies to: 88-88, 151-151

.golangci.yml: 1

12-12: LGTM!

Dockerfile: 2

5-5: LGTM!


58-59: 🔒 Security & Privacy

The comment requests additional runtime validation but does not identify a demonstrated Dockerfile failure or a violated repository validation requirement. The reported absence of a final-image probe does not establish that the image fails to build or that the CA configuration is incorrect.

Dockerfile-58-59: 1

58-59: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Pin the runtime packages consistently.

The command leaves tzdata and wget unpinned. It also uses >= for the CA packages, which permits later versions instead of selecting an exact version. If reproducible runtime images are required, use exact = constraints for all packages and remove the duplicate unversioned ca-certificates entry.

Source: Linters/SAST tools


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 81eee379-0c1c-4276-9ab2-e6b2aa0a0684
📥 Commits

Reviewing files that changed from the base of the PR and between 3a4efbb and 27db6f9.

📒 Files selected for processing: 7
  • .github/workflows/api-contract-tests.yml
  • .github/workflows/code-quality.yml
  • .github/workflows/migrations-validate.yml
  • .github/workflows/tests.yml
  • .golangci.yml
  • Dockerfile
  • go.mod

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

alpine:3.24 still ships libssl3/libcrypto3 3.5.8, and 3.5.9 fixes
High-severity CVEs. Only busybox wget's TLS and apk use OpenSSL in this
image, but the pin costs nothing while the image is being rebuilt anyway.

Ref: ENG-4067

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xernobyl
xernobyl marked this pull request as ready for review October 21, 2026 09:34
@xernobyl
xernobyl added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 5f69443 Oct 21, 2026
10 checks passed
@xernobyl
xernobyl deleted the fix/ENG-4067_go-1.27.3-ca-certificates branch October 21, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants