Skip to content

Add an optional run outcome to results #94

Open

Description

@jamesarosen

Follow-up to #65, from this comment. #88 adds results.errors and notes that "overall-run-outcome questions remain open." This issue is for that question and should not block 0.3.0.

Problem

With results.errors, a report can describe a failed run whose tests all passed, but it still doesn't carry a verdict. Each consumer has to derive one from summary.failed and results.errors, and the framework's own verdict is lost. Playwright, for example, distinguishes failed, timedout, and interrupted.

Precedent

When a format keeps errors out of the test list and still wants the report to carry a verdict, it adds a run-level outcome:

  • Jest: success is false when numRuntimeErrorTestSuites > 0.
  • VSTest TRX: ResultSummary/@outcome="Failed" even when every test passed, with the run-level counters left alone.
  • Cucumber Messages: TestRunFinished.success is required, along with an exception field.
  • node:test: test:summary.success is set false for uncaught errors outside tests, without changing failed (harness.js).
  • pytest-json-report: top-level exitcode.
  • Datadog Test Optimization: session status, which its Playwright integration takes from Playwright's run result.

The exceptions are xUnit v2 and Playwright JSON. Both have an error list but no outcome, which is the shape CTRF 0.3.0 is heading toward.

#65 suggested results.environment.healthy as a stand-in, but no consumer I checked reads it. ctrf-io/github-test-reporter gates only on summary.failed.

Questions to settle

  • Shape. A boolean like Jest's and Cucumber's success, or a status enum? An enum could reuse CTRF's test statuses, or add run-specific values such as timedout and interrupted. A rawStatus-style field could keep the framework's own value, as results.tests[].rawStatus does for tests.
  • Consistency. Should a producer that sets a failing outcome also emit a results.errors entry (see the "dependable signal" issue)? Should a passing outcome be allowed alongside a non-empty results.errors or summary.failed > 0?
  • Merging. Playwright's merge-reports merges run status worst-first: failed, then timedout, then interrupted, then passed. CTRF's merge rule could do the same.
  • Consumers. When present, consumers like github-test-reporter could gate on the outcome directly, instead of deriving one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions