Repository navigation
Description
Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.
Problem
With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.
Precedent
When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:
- Jest:
successis false whennumRuntimeErrorTestSuites > 0. - VSTest TRX:
ResultSummary/@outcome="Failed"even when every test passed, with the run-level counters left alone. - Cucumber Messages:
TestRunFinished.successis required, along with anexceptionfield. - node:test:
test:summary.successis set false for uncaught errors outside tests, without changingfailed(harness.js). - pytest-json-report: top-level
exitcode. - Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.
The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.
#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.
Questions to settle
- Shape. A boolean like Jest's and Cucumber's
success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such astimedoutandinterrupted. ArawStatus-style field could keep the framework's own value, asresults.tests[].rawStatusdoes for tests. - Consistency. Should a producer that sets a failing outcome also emit a
results.errorsentry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-emptyresults.errorsorsummary.failed > 0? - Merging. Playwright's
merge-reportsmerges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same. - Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.
Activity