Repository navigation
feat(error-tracking): build during the run and verify before the report - #424
Merged
Merged
Conversation
Tasks run the project's build after their edits instead of writing code that compiles by construction. A new verify task builds the finished setup before the report, and builds without upload credentials still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
…t shell's limits Tasks build but never run the app (no emulator or simulator), skip commands the shell blocks instead of wrapping them, and treat a missing SDK in the reduced shell as the shell's limit. Python installs no new virtualenv, and init and credentials merge into .env.example instead of overwriting it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu
… settings A package inside a workspace installs with the manager whose lockfile sits at the workspace root, declares the SDK in the manifest that imports it, and gets the CI and deploy files at the repository root wired. CI steps never fail on a missing PostHog secret, configure keeps existing source-map settings and pins, and an Electron main process gets its own capture with a build-time token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu
…steps on the key configure may upgrade a dependency when the install fails on it, not only the build. wire-ci names the condition a posthog-cli pipeline step needs to skip itself when the key is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu
…roject has them init sets up the server the project builds or deploys, not a dev-only script beside it, and capture hooks NestJS through a global exception filter. Server JavaScript that no build compiles gets no upload tasks, and configure never adds a build to create one. install's workspace package-manager rule wins over the skill's own rule, and wire-ci hashes the upload credentials in a Turborepo cache while passing the git variables through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu
hpouillot
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull requests:
Problem
The
error-trackingtasks were told they cannot run the build. Each task had to write code that compiles by construction, and nothing built the finished setup. On Pi the agent can run the project's build (npm run build,cargo build,go build, Gradle, Xcode), so the flow can check its own work. Without that check, runs shipped setups that broke the build:projectId is required when sourcemaps are enabled.posthog-nodedependency that no code imported, and server errors went uncaptured.Changes
verify(new task): runs after every task that edits code, when the upload subgraph is queued. It installs, runs the production build, reads the exit status and the uploader's release and upload lines, and fixes what this run broke. Whenconfigurealready ran a passing build with credentials and nothing edited code after it,verifyreuses that build. It always endsdone, so the report still runs.capture-exceptions: builds or typechecks after its edits and fixes what it broke. It covers the server runtime with the SDK's server library (for Next.js,onRequestErrorininstrumentation.ts) and installs only that library. In a project with no build of its own (most Python, Ruby and PHP apps) it runs nothing: no virtualenvs, package installs or Docker.configure: builds after its edits. The build must pass with and without upload credentials, so the upload skips itself when they are missing. An env-file flag or loader that the project'sstartalready has stays.verifywith the upload subgraph, alongsidewire-ci, and queues the tail one task per turn soreportandtest-setupcan name their dependencies. Python, Ruby and PHP get noverify.install: installs only the SDK the init constructs. The server library is left tocapture-exceptions. A Python project gets no new virtualenv: without one in the project, the package goes into the manifest only.init: does not build, becauseinstallmay still be running beside it. The type-narrowing guidance is gone; the later builds catch type errors.initandcredentials: both add their lines to.env.examplewithedit. They run in parallel, and awritefrom one dropped the other's variables.capture-exceptions,configureandverify: build, never run.expo run:*,react-native run-*andflutter runopen an emulator or a simulator, so an Expo app with generated native folders builds its bundle withnpx expo export --source-maps. A command the shell blocks (make,sh,command -v, a Gradle wrapper outside the root) is skipped and named in the handoff, never wrapped in a new package script. The shell gets a reduced environment, so an Android build there stops atSDK location not found: the task does not retry, and the report asks the user to run that build once themselves rather than set up an SDK. In Go,go vet ./...checks the code without leaving a binary in the project root.report: depends onverifyand reports its build and upload. It writes the file once withwrite;publish_handoffis blocked for it.test-setup: asks first and finishes right away on "no". On "yes" it adds the test affordance and runs the build itself.install,capture-exceptions,configure,verify): whendetect_package_managerfinds no lockfile in the app's directory, the task looks above it for the workspace root's lockfile orpackageManagerfield and uses that manager. npm is used only when no parent has either. This rule wins over the install skill's own rule, which looks at the app's directory only.capture-exceptionsdeclares the server library in the manifest of the package that imports it, including a package nested below the app.wire-ciandlink-releases: in a workspace package they look for the pipeline and deploy files at the repository root. No step fails on an empty PostHog variable, because the user merges the change before creating the secrets; the handoff names the secrets instead. With a Turborepo cache, the upload credentials go in the task's hashedenv, so a build with them never replays an entry made without them, and the git variables go inpassThroughEnv. The cache stays on.configurechanges only what the upload needs. An existing source-map setting (productionBrowserSourceMaps, Vite'sbuild.sourcemap, webpack'sdevtool) stays. A dependency is upgraded only when the install or the build fails on it, and an exact pin stays exact. The upload goes into a build the project already has, here or in a package nested below; a project with none gets no new bundler, compiler or script, and the task reportsnot needed.capture-exceptionscovers an Electron main process with the server library. Its token is read the way electron-vite inlines it at build (import.meta.env.MAIN_VITE_*), because a packaged app has no.env.capture-exceptionshooks NestJS through a global exception filter, the app's existing@Catch()filter when it has one. Nest's filters answer request errors before Express error middleware runs.initsets up each server the project builds or deploys. A script the project only runs in development, with nothing but adevscript, gets no client.sourcemaps.enabledon the upload variables, and guard aposthog-clibuild step on the key.wizard upload-source-mapsreads this skill too.Without the compile-by-construction reasoning the run is also shorter. On
no-posthog-next, averaged over two runs on each side, the run went from 346 s to 298 s andconfigurefrom 102 s to 72 s (22 → 11.5 turns).Test plan
npm test(219 passed) andnpm run build(emitsagents-error-tracking-verify.md).wizard error-trackingruns on Pi with--local-context-millagainst wizard-workbench fixtures in project 89529. After each JS run, a cleannpm ciand a production build, once without the run's.envand once with it:no-posthog-nextverifyreusedconfigure's buildno-posthog-next+ a throwing API routeposthog-nodeinonRequestErrorreact-viteno-posthog-expressno-posthog-sveltekitcicd-github-actions-node-rawdeploy.ymlwired,startkeeps--env-file=.envcicd-github-actions-docker-python-flaskverifycicd-gitlab-ruby-sinatraverifycicd-github-actions-docker-php-laravelverifynuxt-4-3no-posthog-angularno-posthog-vuenode-webpacknode-rollupno-posthog-react-routerno-posthog-honogogo vetandgo buildpass, no blocked commandsrustcargo build --releasepassesios-spmxcodebuildsucceeds,verifyreused itandroidSDK location not foundin the agent's shell, tried oncereact-native-exponpx expo exportpasses, no emulatorreact-nativeandroid/blocked, skippedno-posthog-djangoposthog>=7.59.0inrequirements.txt, no virtualenvStack-specific and real-world apps, on the same setup: the new apps in PostHog/wizard-workbench#4482, and apps from other workbench programs.
no-posthog-nestjs,no-posthog-express-tsc,no-posthog-cloudflare-worker,no-posthog-electron,create-react-appCI=trueand environment variables onlynext-sentryproductionBrowserSourceMapsand the exactposthog-jspin keptno-posthog-fastapi-uvdeploy.ymluv sync --frozen, import without.env, and tests passno-posthog-spring-boot@RestControllerAdvice./gradlew buildpassesvite-yarn-pnpyarnis Yarn 1 and the agent's shell does not allowcorepackaudit/cal-diy,apps/api(Yarn 4 workspace)yarn addfrom the app directory, the SDK declared in the nestedv2package, the root workflow,turbo.jsonanddocker-compose.ymlwiredaudit/cal-diy,apps/webverifyreports the app's ownNEXTAUTH_SECRETbuild requirement as pre-existingcicd-monorepo-pnpm-node-react,backenddeploy-backend.ymlpasses the build args and the key as a build secrettanstack-start-saasastro-ssr-docsflutterios-podfizzy(Rails)Before the workspace and
configurerules,cal-diyapps/apistopped at install withEUNSUPPORTEDPROTOCOL(npm onworkspace:*) and wrote no report, the pnpmbackendleft apackage-lock.jsonand an unwired root workflow, andapps/weblostproductionBrowserSourceMapsand its exactposthog-jspin.A regression pass on the final prompts:
no-posthog-nextreact-viteno-posthog-expresscicd-github-actions-node-rawdeploy.ymlpasses the variables with no failure check.env, and with environment variables onlycicd-github-actions-docker-python-flaskno-posthog-nestjsAllExceptionsFilterfor 5xx, upload in the Docker build before the prune with the key as a build secretno-posthog-express-no-buildaudit/cal-diy,apps/apiv2set up with a global Nest filter, Yarn from the app directory, TurborepoenvandpassThroughEnvwith the cache onverifybuilds@calcom/api-v2Earlier cal.com runs in this pass shaped those rules: one used npm and stopped at install, two set up the dev-only proxy beside
v2(one of them added an esbuild build to it), one turned Turborepo's cache off, and one wired capture throughsetupExpressErrorHandler, which Nest's filters bypass.The Python, Ruby and PHP runs meet the expectations in the workbench's
apps/error-tracking/README.md. Every completed run wrote the report file once, and no task calledpublish_handoff.Before the shell rules above, the same fixtures ran into them:
androidtook 604 s (Gradle retried for about 6 minutes),react-native-expotook 696 s (expo run:androidopened the emulator twice),goandrusttriedmake,shandcommand -v,goleft its binary in the project root, andno-posthog-djangoleft a.venvin the project. One morereact-nativerun on the final prompts stopped when a model call hung for 2 hours andnpm installtimed out; itsconfigureskipped the blocked Gradle wrapper without adding a script.Note
Outside this change:
ANDROID_HOME, so no Android build can run in it, and its fence allows anynpx exposubcommand, includingexpo run:*. Both live in the wizard's shared agent code.$env/static/public, so a SvelteKit build without.envfails with"PUBLIC_POSTHOG_PROJECT_TOKEN" is not exported by "virtual:env/static/public". Switchingexample-apps/sveltekitand the SvelteKit init guidance to$env/dynamic/publicwould fix it.🤖 Generated with Claude Code
https://claude.ai/code/session_01ShmV3MZAFkwMroJi6AiBmu