Skip to content

[TokenSpeed] Add B300 Qwen3.8 AgentX nightly benchmark / [TokenSpeed] 新增 B300 Qwen3.8 AgentX nightly 配置 - #5116

Open
tuanzhangCS wants to merge 2 commits into
mainfrom
feat/qwen38next-b300-tokenspeed-nightly-agentx
Open

tuanzhangCS wants to merge 2 commits into
mainfrom
feat/qwen38next-b300-tokenspeed-nightly-agentx

Conversation

@tuanzhangCS

@tuanzhangCS tuanzhangCS commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add B300 TokenSpeed AgentX coverage for nvidia/Qwen3.8-Flash-Next-NVFP4, using the workload and per-concurrency tuning from #4922. The new qwen3.8next-fp4-b300-tokenspeed-agentic-mtp entry runs TP1/EP1 on cluster:b300-dsxe at concurrency 1, 2, 4, 8, 20, 36, 56, 80 and 108. Its running-request/decode-graph caps are 104, 104, 104, 2, 6, 12, 35, 72 and 104, respectively, with a 3600-second profile.

Pin the amd64 CUDA 13 / PyTorch 2.14.0 / FlashInfer 0.7.4 runner by digest and the SHA256-verified 20261117 wheels: TokenSpeed 0.1.1.post20261117, kernel 0.1.5.post20261117, engine 0.1.1.post20261117, scheduler 0.1.58 and SMG 1.16.0.post20261115. Keep checkpoint revision fc694b54fb0174e0913e6adf86691ef85a4ead47, its embedded MTP head and checkpoint-selected quantization, FP8 E4M3 target/draft KV, BF16 SSM state, three MTP steps, 262144-token context and 16384-token prefill chunks. The 236 GB Host L2 cache and 4 GB retraction snapshot pool retain the locally exercised 241 GB CPU DRAM budget per GPU, selected through dram-utilization: 0.645.

The existing native srt-slurm backend provides headless TokenSpeed workers, SMG ZMQ routing and HTTP metrics. The recipe omits the wheel's removed --enable-drafter-prefill-graph option; target prefill graphs remain enabled through 8192 tokens and supported drafter extend graphs are captured automatically. Synthetic throughput acceptance continues to come from the committed golden curve, while eval-only runs use real verification.

Coverage and limitations: pre-submission execution covers only local BS1, BS8 and BS80 with these wheels. The other six concurrency points, the newly pinned CI container and the complete native CI path have not yet been exercised together. All three local runs include a shared approximately two-minute stall of unknown origin, retained in the results; these measurements do not establish a nightly performance improvement or regression.

Validation details

The three local B300 native AgentX runs each completed a 3600-second profile and passed native validation, with zero exported request errors. Successful requests: BS1 701, BS8 1510, BS80 11785. BS80 canceled four in-flight requests at the profile/grace cutoff. The local run used an isolated launcher adapter; this PR contains the benchmark configuration and uses the repository's existing native CI launcher.

AI model disclosure
  • Model/version: The runtime did not expose a verifiable exact model identifier; the exact model/version could not be verified.
  • Role: An AI coding assistant prepared the recipe, master configuration, documentation and changelog, ran the local benchmarks and checked the generated configuration. No delegated agents contributed.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, a maintainer with write access to the repository has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

参考 #4922 新增 B300 的 Qwen3.8-Flash-Next-NVFP4 TokenSpeed AgentX 配置。在 cluster:b300-dsxe 上使用 TP1/EP1,覆盖并发 1、2、4、8、20、36、56、80、108,对应最大运行请求数和 decode graph 上限为 104、104、104、2、6、12、35、72、104,每档 profile 3600 秒。

固定 20261117 的 TokenSpeed、kernel、engine wheel 及 SHA256,使用 FlashInfer 0.7.4 的 amd64 容器并固定 digest,同时固定 scheduler 0.1.58 和 SMG 1.16.0.post20261115。保持相同模型 revision、模型随附的 MTP head 和 checkpoint 所选量化、FP8 target/draft KV、BF16 SSM、三步 MTP、262144 context、16384 prefill chunk,以及 236 GB Host L2、4 GB snapshot pool 和每 GPU 241 GB CPU DRAM 预算。复用原生 srt-slurm 的 ZMQ 服务和指标采集,移除新版 wheel 已删除的 drafter prefill graph 参数。吞吐测试从 golden curve 推导 synthetic acceptance,eval-only 使用真实验证。

本地实测仅覆盖 BS1、BS8、BS80,三档均完成 3600 秒 profile、通过原生验证且导出请求错误为零;BS80 在 profile/grace 截止时取消了四个进行中的请求。其余六档、新固定的 CI 容器及完整 CI 路径尚未联合实测。三档结果均保留一次约两分钟、原因未明的共同停顿,不能据此判断 nightly 性能提升或回退。本地使用的隔离 launcher adapter 未包含在本 PR 中。

改动类型为配置和文档。AI 助手完成配置、文档、changelog、本地 benchmark 和验证;运行环境未提供可核实的精确模型标识,因此无法确认完整模型名称/版本。无委派 agent 参与。

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@tuanzhangCS tuanzhangCS added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 17, 2026 — with ChatGPT Codex Connector
Comment thread inferencex-e2e/perf-changelog.yaml Resolved
@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant