Skip to content

fix(runner): preview telemetry accuracy for Tier 1 and Tier 2 (DEV-3145) - #395

Merged
demtario merged 2 commits into
masterfrom
fix/DEV-3145-preview-telemetry-accuracy
Sep 30, 2026
Merged

demtario merged 2 commits into
masterfrom
fix/DEV-3145-preview-telemetry-accuracy

Conversation

@demtario

@demtario demtario commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Context

Follow-ups from #371 on how the editor's live preview reports compile and runtime errors (ClickUp DEV-3145). After the Sentry trim, sandpack.compile_error and preview.runtime_error are the only record of handled preview errors, so their accuracy matters. This PR covers all six items on the ticket:

  1. Tier 2 reporter slots (fixed). Only Tier 1 reset the in-preview reporter's 20-error budget. Vite HMR keeps the injected document, so a Tier 2 page went silent after 20 distinct errors. ContainerRuntime.flush() now posts the same reset (shared postMonitorReset helper in monitor.ts) before writing files, and re-arms the stderr relay budget. stderrSeen is not cleared, so old status-log lines are not re-relayed.
  2. Compile errors shared the runtime ceiling (fixed). demoEventCollapse now counts compile: keys against their own DEMO_COMPILE_CEILING, so neither kind can starve the other.
  3. bucket on sandpack.compile_error (fixed). Added to the registry row and contract §5, and emitted from a getter that is read when the error arrives (the collapse may emit up to 2 s later, after a bucket switch). Ingest already mapped bucket, covered by a new Faro normalise test. ADR-0041 still opens with "whether the last release raised compile errors on the next bucket" and no doc or commit shows the omission was deliberate. The next question itself is unchanged: alerts and dashboards still group by ht_major only.
  4. Stale preview after a Tier-1 syntax error (fixed, minimal). SandpackRuntime tracks whether the newest push failed to transpile and exposes onStaleChange. Once the edit burst settles and the preview is ready, the pane gets data-preview-stale="true" and the status bar reads "Showing last successful run". No new Sentry event and no error card. An older push's bundler start does not clear a newer failure.
  5. vue-cli late bundler error (no fix). Wrote the regression test for the suspected order. The order "error arrives between the first run's start and the newer run's start" passes on current code. The order start(A), start(B), error(A) cannot be told apart from B's own bundler rejection, which an existing test requires to suppress stale relays, so a fix there would break that. Left as a test only.
  6. e2e stubShell (fixed). New abortSandpackHosts in e2e/helpers.ts matches the versioned *-sandpack.codesandbox.io host. stubShell and the 20 inline or local copies of the two-glob abort across 18 specs now use it. The E2E_LIVE-gated specs and the ready-state cases that rely on the live bundler are untouched.

Types of changes

  • New example
  • Update to an existing example
  • README / documentation change
  • Demo runner (runner/) change
  • CI / tooling change

How was this verified?

  • Raw pnpm typecheck clean and pnpm test in runner/: 2567 tests, 2567 pass, 0 fail (re-run by me, not only by the implementing agents).
  • Every fix has a test that fails without it (edited the fix out, saw the failure, restored): container reset, monitor reset, compile ceiling, stale lifecycle, older-start guard, eager bucket, registry bucket blob, Faro ingest.
  • Playwright e2e on a dedicated preview port with the authoring app built: 175 passed, 50 skipped (env-gated), 0 failed.
  • Not covered by an automated test: the App.tsx / EditorShell wiring of data-preview-stale and the status bar note.
  • Not verified: the Tier 2 fix against a live container, and item 5 against the live vue-cli bundler.
  • Code review (high effort) ran on the first commit; its valid findings are fixed in the second commit. Not changed: the per-flush Tier 2 reset matches what Tier 1 does per push, and the app-level collapse ceiling still bounds what reaches Sentry.

Checklist

  • New/renamed example: added to runner/config/frameworks.json (see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.com
  • New example: added a row to the tables in README.md
  • Ran pnpm build (and pnpm dev) in the affected example/server-example locally

The checklist items are for new or renamed examples; this PR changes only runner/.

Related issue(s):

  1. ClickUp DEV-3145

Note

Medium Risk
Touches live-preview error reporting, Sentry/metric collapse, and Tier-2 monitor resets; behavior is heavily tested but affects observability and what users see during broken edits.

Overview
Improves preview error telemetry and UX after compile/runtime reporting became the main signal for handled preview faults.

Tier 1 (Sandpack): When the latest edit fails to transpile but the iframe still shows the last good run, the runtime exposes onStaleChange; the authoring app debounces that with the same edit-settle window as other burst logic, then sets data-preview-stale on the preview pane and “Showing last successful run” on the status bar (only while status is ready). Stale clears on a successful rerun start or an unchanged push, without treating an older bundler start as fixing a newer failure.

Telemetry: sandpack.compile_error now includes an optional bucket (read at error time so collapsed emits stay correct). Compile vs runtime collapse uses a separate ceiling for compile: keys so one stream cannot exhaust the other’s budget.

Tier 2 (container): On each post-ready edit flush, the runtime re-arms the in-preview monitor (shared postMonitorReset) before writing files and resets the stderr relay budget without re-relaying already-seen log lines—matching Tier 1’s per-run reset under Vite HMR.

E2E: abortSandpackHosts centralizes aborting versioned *-sandpack.codesandbox.io hosts so deterministic specs do not accidentally hit the live bundler.

Docs/registry/tests updated for the new metric dimension and behaviors.

Reviewed by Cursor Bugbot for commit 5e74d28. Bugbot is set up for automated code reviews on this repo. Configure here.

demtario and others added 2 commits September 30, 2026 12:15
- Tier 2 posts the reporter budget reset on every flush and re-arms the stderr relay budget
- sandpack.compile_error gets its own per-page ceiling instead of sharing the runtime one
- sandpack.compile_error carries a bucket attribute (registry, contract, emit, ingest test)
- Tier 1 exposes a settled stale-preview state (data-preview-stale, status bar note) after a syntax error
- regression test for a superseded bundler error between pushes
- e2e stubShell and its inline copies also abort the versioned sandpack host

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- an older push's start no longer clears a newer transpile failure's stale state
- compile_error reads the bucket when the error arrives, not when the collapse emits
- share the compile key prefix between the collapse and sentry.ts
- show the stale note only while the preview is ready

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@demtario demtario self-assigned this Sep 30, 2026
@demtario
demtario merged commit 29c487a into master Sep 30, 2026
10 checks passed
@demtario
demtario deleted the fix/DEV-3145-preview-telemetry-accuracy branch September 30, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant