Skip to content

[Epic] Readable benchmark PR comments for multi-case runs #355

Open
0 of 1 issue completed
Open

Description

@soroushm

Epic Overview 🎯

bench-action's PR comment doesn't scale past a couple of bench files. soroush-tech/umber-style#57 shows the problem with 7 files: 7 headings, 7 tables and 14 rows. The number the gate actually checks, speed vs baseline, never appears. The table shows "vs fastest", but which row is fastest changes from file to file, so you can't scan the comment for a regression. Case labels also repeat the file name and the case keys (core: merging five classes :: previous).

This epic replaces it with a single summary table, one row for each non-baseline case compared against the baseline. The per-file detail is folded below it. It also makes the comment configurable, so more than one bench job can post on the same PR.

Implementation Details 🛠️

  • Data source (@soroush.tech/bench 2.3.0): a new --json-file writes each case's raw numbers (ns, bytes) and its ratioPct against the baseline, in the same container-mount pattern as --md-file. The action reads this instead of parsing the formatted markdown, which has lost precision.
  • Summary layout (bench-action 1.2.0):
    ## ✅ Benchmarks — 7 of 7 passed
    Baseline `previous` · each case must reach 80% of baseline speed · 1 round
    
    |    | benchmark                    | case  | time      | baseline  | speed | alloc  |
    | :-: | ---------------------------- | ----- | --------: | --------: | ----: | -----: |
    | ✅ | a theme and its variables    | local | 1.35 ms   | 1.21 ms   | 90%   | +29.5% |
    | ✅ | extracting the CSS of a page | local | 123.31 µs | 139.50 µs | 113%  | 0.0%   |
    
    <details><summary>Full results (7 files)</summary> …per-file tables… </details>
    
    There is one row per non-baseline case, so a file comparing N packages gives N − 1 rows under the same benchmark name. Failed rows come first, then the rest from slowest to fastest.
  • New inputs: comment-details (collapsed by default, or open or none) controls the folded per-file tables. comment-title sets the heading and is also part of the hidden sticky-comment marker, so two bench jobs on one PR keep separate comments. Leaving both inputs out gives the new layout with today's single sticky comment. Both the branded relay path ([Epic] Bench comment-relay worker at api.bench.soroush.tech #309) and the token path render the same body.
  • Rollout: release bench 2.3.0 → re-vendor it in bench-action → release bench-action 1.2.0 → bump umber-style to check the comment on a real PR.

Key Features ✨

Feature Description
JSON results soroush-bench --json-file writes unformatted per-case numbers plus ratioPct against the baseline, for any number of cases.
Summary table One table across all files and one row per non-baseline case against the baseline, showing the gated speed %. Failures come first.
Folded detail Today's per-file tables move into a <details> block. comment-details controls it: collapsed, open or none.
Comment title comment-title sets the heading and the sticky marker, so several bench jobs can comment on one PR without overwriting each other.

Acceptance Criteria ✅

Dependencies

Metadata

Metadata

Assignees

Labels

area: benchAffected areaepicA large body of work that can be broken down into smaller tasks or stories.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions