fix(o11y): box and drain robustness follow-ups (DEV-3144)

[demtario]

Context

Follow-ups from DEV-3144 (PR #371 / ADR-0041) for the sleeping Grafana box and the R2 inbox drain. This PR closes the items that were verifiable and safe to batch; the Loki index_ schema change is in its own PR because it has a deploy-date constraint. Per item: /telemetry/collect now has the same 10 s ingest deadline as the other ingest routes (503 + Retry-After), and a commit that lands after the deadline still writes its accepted items' analytics points once. GrafanaBox guards the constructor-time outbound interception failure that the SDK leaves as an unhandled rejection, reports fail-open through a new o11y.ae_degraded metric and an ae-outbound-degraded alert rule (2 h window), resumes draining after a spend-cap pause while a visitor holds the box, and backs off and gives up when the box stays not-ready. Symbolication lists existing source maps once per service.version before reading, so forged paths no longer spend an inbox object's map budget, and a transient map read or listing failure defers only that key (bounded to keys under 6 h old) instead of pushing it unsymbolicated. Deferred keys are excluded for the rest of the wake so they no longer starve later keys. Local dev checks the clean-shutdown marker in MinIO over a SigV4 HEAD, so the ledger commits locally. A "dropped by reason" panel is added to Observability self. A nightly workflow builds the box image and runs stop-roundtrip.mjs, and the o11y-local E2E API boot wait is configurable and fails fast.

Not in this PR (decided during triage): the lost-reply accepted undercount (not fixable without changing dedupe semantics, stays documented in ADR-0041), ClickHouse empty header names (unproven, needs a run against plugin 3.5.0), splitting the InboxWriter per tenant, bounding the stop grace period, the @cloudflare/containers "Network connection lost" (upstream report only), and an idle-stop integration check.

Types of changes

• New example
• Update to an existing example
• README / documentation change
• Demo runner (runner/) change
• CI / tooling change

How was this verified?

Every behavior change was written test-first and revert-checked (the new test fails against the previous code, passes with the change). Full pipeline suite from runner/pipeline: node --experimental-strip-types --test *.test.mjs is 2669 tests, 2669 pass, 0 fail, 0 skipped. npx tsc --noEmit in runner/workers/o11y is clean and pnpm typecheck in runner passes for all packages. actionlint is clean on the new workflow, and its commands (docker compose build box + stop-roundtrip.mjs) ran locally with all checks passing. Each piece had a low-effort review, then a high-effort review of the whole branch whose findings (a workflow that would have failed at setup-node, drain starvation by deferred keys, lost late-commit analytics points, plus smaller items) are fixed here. Not verified: the new workflow on GitHub, the o11y-local E2E end to end, the dashboard panel rendered in Grafana, and the MinIO SigV4 HEAD against a real MinIO (the signer matches the AWS test vector; please run one o11y-dev wake). The new alert and metric go live only after the o11y worker deploys.

Checklist

• New/renamed example: added to runner/config/frameworks.json (see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.com
• New example: added a row to the tables in README.md
• Ran pnpm build (and pnpm dev) in the affected example/server-example locally

None of the example checklist items apply; this PR changes only runner/ and CI.

Related issue(s):

1. DEV-3144 (https://app.clickup.com/t/123kvxebzp5)

---

Note

Medium Risk
Touches core o11y ingest deadlines, GrafanaBox lifecycle, drain ledger ordering, and telemetry accounting; behavior is heavily covered by pipeline tests but production box/stop and MinIO marker paths are not fully exercised in CI.

Overview
Hardens the sleeping Grafana box, inbox drain, and browser ingest paths after DEV-3144, with broad test and doc updates.

Ingest: /telemetry/collect now uses the same 10 s InboxWriter.ingest deadline as other routes (503 + Retry-After). Analytics points for inbox-backed items are written only for accepted outcomes; late commits after a timeout are finished via ctx.waitUntil so metrics are counted once when the client retries as duplicate.

GrafanaBox / drain: Fail-open or failed reload of the ae.internal outbound interception emits o11y.ae_degraded and feeds a new ae-outbound-degraded alert (2 h window). Constructor-time interception rejections are handled so they are not unhandled rejections. While spend-cap pauses drains, an awake box with a visitor keeps a ~60 s recheck chain and resumes draining when the pause clears. Boxes that stay not-ready for 10 minutes back off, emit a drain error, and stop when quiet (visitors keep polling). Failed inbox or source-map reads defer keys for the rest of the wake (nextWrittenKeys skips them) so they do not starve the batch; only-deferred backlog can slow-recheck or stop like an empty backlog. Symbolication lists maps per service.version before GETs (caps forged paths), retries transient map/list failures, and defers young keys (<6 h) instead of pushing unsymbolicated bodies. Local wake resolution checks clean-shutdown markers via SigV4 HEAD to MinIO (marker.ts).

CI / observability: New o11y-box-nightly.yml builds the box image and runs stop-roundtrip.mjs. Observability self gains an o11y.ingest dropped-by-reason panel. E2E o11y-local shares wait-for-server with configurable API boot timeout and fast-fail when the child exits.

^{Reviewed by Cursor Bugbot for commit 47bb325. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 24fc828. Configure here.}