Repository navigation
[TokenSpeed] Add B300 Qwen3.8 AgentX nightly benchmark / [TokenSpeed] 新增 B300 Qwen3.8 AgentX nightly 配置 - #5116
Open
tuanzhangCS wants to merge 2 commits into
Open
tuanzhangCS wants to merge 2 commits into
tuanzhangCS wants to merge 2 commits into
Conversation
tuanzhangCS
requested review from
Ankur-singh,
RohitNagraj,
kedarpotdar-nv,
nvpohanh and
wzhao18
as code owners
November 17, 2026 09:26
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=44055689902 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=44055689902 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add B300 TokenSpeed AgentX coverage for
nvidia/Qwen3.8-Flash-Next-NVFP4, using the workload and per-concurrency tuning from #4922. The newqwen3.8next-fp4-b300-tokenspeed-agentic-mtpentry runs TP1/EP1 oncluster:b300-dsxeat concurrency 1, 2, 4, 8, 20, 36, 56, 80 and 108. Its running-request/decode-graph caps are 104, 104, 104, 2, 6, 12, 35, 72 and 104, respectively, with a 3600-second profile.Pin the amd64 CUDA 13 / PyTorch 2.14.0 / FlashInfer 0.7.4 runner by digest and the SHA256-verified 20261117 wheels: TokenSpeed
0.1.1.post20261117, kernel0.1.5.post20261117, engine0.1.1.post20261117, scheduler0.1.58and SMG1.16.0.post20261115. Keep checkpoint revisionfc694b54fb0174e0913e6adf86691ef85a4ead47, its embedded MTP head and checkpoint-selected quantization, FP8 E4M3 target/draft KV, BF16 SSM state, three MTP steps, 262144-token context and 16384-token prefill chunks. The 236 GB Host L2 cache and 4 GB retraction snapshot pool retain the locally exercised 241 GB CPU DRAM budget per GPU, selected throughdram-utilization: 0.645.The existing native srt-slurm backend provides headless TokenSpeed workers, SMG ZMQ routing and HTTP metrics. The recipe omits the wheel's removed
--enable-drafter-prefill-graphoption; target prefill graphs remain enabled through 8192 tokens and supported drafter extend graphs are captured automatically. Synthetic throughput acceptance continues to come from the committed golden curve, while eval-only runs use real verification.Coverage and limitations: pre-submission execution covers only local BS1, BS8 and BS80 with these wheels. The other six concurrency points, the newly pinned CI container and the complete native CI path have not yet been exercised together. All three local runs include a shared approximately two-minute stall of unknown origin, retained in the results; these measurements do not establish a nightly performance improvement or regression.
Validation details
The three local B300 native AgentX runs each completed a 3600-second profile and passed native validation, with zero exported request errors. Successful requests: BS1 701, BS8 1510, BS80 11785. BS80 canceled four in-flight requests at the profile/grace cutoff. The local run used an isolated launcher adapter; this PR contains the benchmark configuration and uses the repository's existing native CI launcher.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied./use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
参考 #4922 新增 B300 的
Qwen3.8-Flash-Next-NVFP4TokenSpeed AgentX 配置。在cluster:b300-dsxe上使用 TP1/EP1,覆盖并发 1、2、4、8、20、36、56、80、108,对应最大运行请求数和 decode graph 上限为 104、104、104、2、6、12、35、72、104,每档 profile 3600 秒。固定 20261117 的 TokenSpeed、kernel、engine wheel 及 SHA256,使用 FlashInfer 0.7.4 的 amd64 容器并固定 digest,同时固定 scheduler 0.1.58 和 SMG 1.16.0.post20261115。保持相同模型 revision、模型随附的 MTP head 和 checkpoint 所选量化、FP8 target/draft KV、BF16 SSM、三步 MTP、262144 context、16384 prefill chunk,以及 236 GB Host L2、4 GB snapshot pool 和每 GPU 241 GB CPU DRAM 预算。复用原生 srt-slurm 的 ZMQ 服务和指标采集,移除新版 wheel 已删除的 drafter prefill graph 参数。吞吐测试从 golden curve 推导 synthetic acceptance,eval-only 使用真实验证。
本地实测仅覆盖 BS1、BS8、BS80,三档均完成 3600 秒 profile、通过原生验证且导出请求错误为零;BS80 在 profile/grace 截止时取消了四个进行中的请求。其余六档、新固定的 CI 容器及完整 CI 路径尚未联合实测。三档结果均保留一次约两分钟、原因未明的共同停顿,不能据此判断 nightly 性能提升或回退。本地使用的隔离 launcher adapter 未包含在本 PR 中。
改动类型为配置和文档。AI 助手完成配置、文档、changelog、本地 benchmark 和验证;运行环境未提供可核实的精确模型标识,因此无法确认完整模型名称/版本。无委派 agent 参与。