This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
- Fork the repository and create your branch from
main. - Install dependencies:
npm install - Build:
npm run build - Run tests:
npm test - Lint:
npm run lint - If you've added code, add tests that cover your changes.
- Ensure the test suite passes and linting is clean.
- Submit a pull request.
Please use GitHub Issues to report bugs or request features. Before filing a new issue, please check if one already exists.
Use Node 22 (the CI version) or a newer supported Node release. Install the locked dependencies
with npm ci. The existing database integration tests also require the sqlite3 command on
your PATH. There are three complementary test layers:
| Layer | Command | What it verifies |
|---|---|---|
| Unit and fixture tests | npm test |
Parsers, rules, calculations, and synthetic-data invariants |
| Browser webview tests | npm run build then npm run test:e2e |
Dashboard interactions with mocked host responses |
| Packaged VS Code tests | npm run test:vscode |
Installation, activation, workers, real webview messaging, navigation, reload, and empty data |
Before the first browser run, install its browser with npx playwright install chromium.
On a Linux machine without a display, install the desktop dependencies with
npx playwright install-deps chromium, then run xvfb-run -a npm run test:vscode.
Windows and macOS run the same npm command without the virtual display.
npm run test:vscode compiles the test code, packages the extension through the normal packaging
script, and installs that VSIX into a disposable VS Code installation. It never installs into
your normal editor. To test an already-built artifact without rebuilding it, use:
npm run test:vscode -- --vsix ai-engineer-coach-0.1.0.vsixSubstitute the filename produced by npm run package when the package version changes.
The runner records the package's SHA-256 hash and checks that the artifact did not change.
CI and releases pass the exact package path this way; publishing happens only after the
browser and desktop suites pass, and the release uses that same VSIX.
By default, desktop tests use the minimum version in package.json's engines.vscode.
Use --vscode-version 1.125.0 to select an explicit compatible version, or
--vscode-version latest to check the current stable release. An incompatible pinned version
fails before launching. The compatibility workflow
runs the minimum and latest versions on Linux, Windows, and macOS weekly, on manual dispatch,
and on PRs that change the desktop harness, its fixtures, or the shared E2E action.
The desktop runner uses ExTester. Its actively tested editor-version window moves with new releases; older versions are best-effort. Keep the minimum-version tests rather than silently increasing our public engine requirement to accommodate the runner. Dependency updates use the existing Dependabot workflow.
The checked-in data is tests/vscode/data/projects.json:
fictional project names, prompts, source code, tests, model IDs, and a seven-day activity schedule.
It is entirely synthetic, not sampled or anonymized from anyone's personal logs. Three projects
cover TypeScript (inventory-api), Python (sensor-pipeline), and Go (release-worker).
tests/vscode/fixtures.ts materializes those definitions into raw
chatSessions and chatEditingSessions JSON in the platform's normal discovery layout,
entirely inside a fresh temporary home. Initially there are 21 sessions, 63 requests, 315 added
lines and 294 net lines across seven active days, including weekday, weekend and evening activity.
Each session records file creation, a one-line correction, and test creation; the same code also
appears in responses to exercise edit/code-block deduplication. The reload test adds one session,
then verifies 22 sessions, 66 requests and 330 added lines, including a second reload without
duplicates. The empty-data scenario starts a separate process with no session sources.
The data is varied but deterministic rather than randomly changing on every build. The most recent activity day is the day before the run, with earlier days and local hours taken from the schedule, so charts remain populated. The unit suite checks exact code totals, all seven heatmap days, overlapping Timeline sessions, and the three project summaries through the real parser and analyzers. Desktop tests do not replace the parser, workers, VS Code API, or message bridge. They also check that input log files remain byte-for-byte unchanged except for the test's explicit addition. Personal home, AppData, Copilot data, temporary directories, extension directories, and VS Code profiles are isolated; no sign-in or live model call is required.
Only editor/driver download archives are cached under .vscode-test/cache. Each invocation
creates fresh profiles, fixtures, and application caches and removes them after the run.
Diagnostics remain under test-results/vscode, including runtime and driver logs, failure
screenshots, and a package-hash summary. Browser traces, screenshots, and the HTML report
are stored separately under test-results/browser and test-results/browser-report.
GitHub Actions uploads these diagnostics even when a test fails.
The pipeline logs the generated counts and includes a Synthetic test data section in its
summary and gallery. Downloaded artifacts include each suite's dataset.json, the checked-in
definitions under fixtures/projects.json, the actual initial input copies under
fixtures/initial, and the planned additional files under fixtures/reload. These are generated
test inputs, not user logs; the reload files are kept separate from the initial dataset.
Successful milestones also capture real VS Code screenshots: dashboard totals and activity, Timeline
session details, nonzero Output by project, Patterns Work Hours and Projects, reload, reopening,
and the expected empty-data state. In a GitHub Actions
run, open the Summary and follow Download screenshots and test diagnostics. Extract the
artifact, then open vscode/<run>/gallery.html; its images are embedded, so it works offline.
The gallery includes captions, suite outcomes, dataset counts, the editor target, and the tested VSIX hash.
Individual PNGs are in each suite's snapshots directory. Locally, the same gallery is under
test-results/vscode/<run>/gallery.html.
These screenshots document the synthetic-data run, not pixel-comparison baselines. The capture helper waits for chart pixels to settle so it does not show an in-progress animation. Captures are preserved even if a later assertion fails; the report still marks that suite as failed. Existing automatic failure screenshots remain separate.
Add the regression test in the same PR as the feature or fix. Prefer the smallest layer that can reproduce the behavior: parser/calculation changes belong in Vitest, dashboard interaction changes in Playwright, and packaging or host/webview boundary changes in the desktop suite. Do not duplicate every browser scenario in VS Code.
Add lowercase, hyphenated *.spec.ts files directly under
tests/vscode/suite. The runner discovers new source suites automatically;
each file gets a separate process, profile, and baseline synthetic data. empty.spec.ts is the
no-data exception. Reuse the Dashboard helper and small synthetic
fixtures. Keep selectors in the helper, wait for observable readiness instead of fixed sleeps,
and assert exact results.
After a meaningful assertion, use await dashboard.snapshot('01-feature', 'What the screenshot demonstrates')
to add a named milestone to the gallery. Use a unique lowercase, hyphenated name within each suite.
The desktop suite is sequential, has no automatic test retries, and rejects focused, skipped, or
empty test runs. Inherited MOCHA_GREP and MOCHA_INVERT filters are removed so a nonempty subset
cannot pass as the full suite. Reload replaces the webview's frames, so the dashboard helper waits
for a different document and reacquires detached frames within a single deadline, including
ExTester's wrapped stale-outer-frame error. It does not repeat the reload command or suppress
unexpected driver failures.
Its TypeScript and fixture tests are included in npm run check.
Use the shared E2E action for pipeline changes so PR, release, and compatibility runs do not acquire different setup procedures. Investigate failures using the saved screenshots and logs; do not bypass the release gate or disable an operating-system security feature globally to get a green run.
If you discover a security vulnerability, please follow the instructions in SECURITY.md. Do not report security vulnerabilities through public GitHub issues.
Detection rules and metrics are the primary extensibility surface of AI Engineer Coach. Each one is
a self-contained markdown file with YAML frontmatter and a small DSL — no code changes required to
ship a new one. Built-in rules live in src/core/rules/ and metrics in
src/core/metrics/.
See docs/AUTHORING_RULES.md for the full authoring guide: file format, annotated rule and metric examples, the local testing workflow, and links to the DSL reference.