Repository navigation
feat(bench): progress lines per launch and outcome - #144
Merged
Merged
Conversation
The MR6-v2 re-bench hid a hung challenge for ~40 min: the protocol pipes the bench output through tail -45 and the trails stayed silent, so nothing showed the oldest in-flight run. run_bench_lab now prints one flushed progress line per (model, rep, challenge) launch and one per outcome - FLAG|miss with the elapsed seconds - making a stuck run visible in the bench log itself. benchmark.md gains the ronin-cache lesson: the installed ronin cache is the SSOT of lab execution - land the exploit against every target class before rollouts/benches, or the breakage poisons everything downstream (the v2 measured 0/30 uaf rollouts from one menu mismatch).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The MR6-v2 re-bench hid a hung challenge for ~40 min: the protocol
pipes the bench output through
tail -45and the trails stayedsilent, so nothing in the log showed the oldest in-flight run (the
diagnosis took 40 min and three orphaned ruby workers 1-3h old).
Changes
run_bench_labprints one flushed progress line per(model, rep, challenge) launch and one per outcome:
[bench base rep 3/10] ctf-lictor-uaf_train_b-001: FLAG (elapsed 84s).A stuck run is now visible in the bench log itself.
docs/explanation/benchmark.md: the ronin-cache lesson - theinstalled ronin cache is the SSOT of lab execution; land the exploit
against every target class before rollouts/benches (one menu
mismatch measured 0/30 rollouts and poisoned datasets + retrain).
Checklist
TestBenchProgressLine), then minimal fixtests/unit/test_bench_lab.py+test_final_bench.pygreen