Repository navigation
Define the large-asset benchmark corpus - #47
Merged
Merged
Conversation
ru_maxrss starts from the parent's resident size at fork and survives exec, so a benchmark launched by a large process reported the parent's memory as part of its own peak. VmHWM is the process image's own high-water mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
tests/benchmark/corpus.json declares four tiers: the committed fixtures, the committed CC0 subsets, a generated scale ladder (1,048,576, 4,194,304, and 7,995,392 Gaussians) in PLY, SPZ, and SOG, and real exports kept outside the repository, plus provisional resource targets. scripts/benchmark_corpus.py generates the ladder deterministically by tiling the committed CC0 yashica-t4-top8192.ply on a grid, encodes all three formats from the same rows, checks the output against recorded hashes, and measures any tier through benchmark_import.py one process per asset. The SOG fixture generator gains a planar VP8L encoder that emits the reference encoder's bytes for multi-million-texel planes; the committed fixtures regenerate unchanged. The composed root build registers benchmark_corpus_manifest (manifest, committed hashes, encoder equivalence) and benchmark_corpus_ci (fixture, corpus, and 1M rows against the targets), so the workspace CTest cells run both on every hosted platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the next v0.6.0 immediate action in
docs/roadmap/current.md: define the large-asset benchmark corpus.tests/benchmark/corpus.jsondeclares four tiers, each with a recorded license:fixture: the committed synthetic fixtures, one per format.corpus: the six committed CC0 8,192-Gaussian subsets.scale: a generated ladder of 1,048,576 / 4,194,304 / 7,995,392 Gaussians in PLY, SPZ, and SOG. The top step is the largest whole number of tiles under the 8M import ceiling.real: full-size exports kept outside the repository. These are the four retained CC0 source exports, the local Postshot cactus files, and the Graphdecogarden/bicycle, which are download-on-demand and non-commercial.The manifest also sets provisional resource targets: metadata-only read and
CanReadat most 0.25 s, and peak resident memory at most 0.35 GiB + 0.60 GiB per million Gaussians.scripts/benchmark_corpus.pyhas four commands:listandverify.generate: tiles the committed CC0yashica-t4-top8192.plyon a grid and encodes all three formats from the same rows. It checks the output against the recorded hashes: file SHA-256 for PLY and SOG, and the decompressed-payload SHA-256 for SPZ.run: measures each asset throughbenchmark_import.py, one process per asset, and gates the result on the expected count, SH degree, and targets.The SOG fixture generator gains
webp_lossless_planar. It produces the same VP8L bitstream as the reference encoder, built from per-channel bytes so it works on multi-million-texel planes. The committed fixtures regenerate byte-identical.CTest, composed root build only:
benchmark_corpus_manifestrunsverify, andbenchmark_corpus_cimeasures the fixture, corpus, and 1M rows. Both run in theworkspace-ctest-pr-*cells.Fix: on Linux,
benchmark_import.pynow reads peak memory fromVmHWM.ru_maxrssstarts from the parent's RSS at fork and survives exec, so it inflated the peak when a large parent launched the benchmark.Docs: new
docs/reference/BENCHMARK_CORPUS.md. The docs index,BUILDING.md,WORKSPACE.md,PERFORMANCE_BASELINES.md, the roadmap, the backlog, and the CHANGELOG are updated.Why?
The roadmap's step 2 is to "define and measure large-asset validation targets". No redistributable real asset reaches millions of Gaussians, so this adds a deterministic generated ladder up to the import ceiling. It sits beside the real references that design policy §12.1 names.
Validation
python scripts/benchmark_corpus.py verify: 26 assets, 0 problems. The planar encoder matches the reference encoder at four image sizes.generate --tier scale: all nine assets generated in about 2 minutes. A secondgenerate --ciinto a fresh directory reproduced the recorded hashes.CheckCloudContractviolations, and decoded positions agree across formats within quantization.plugins/gaussian-sog/tools/generate_fixtures.pyregenerates the fixtures with no diff.runplumbing (gating, report, temp-dir cleanup) was exercised against a stubpxr. OpenUSD is not available in this environment, so the realbenchmark_corpus_cimeasurement is left to the hosted workspace CTest cells on this PR.Checklist
Reviewer notes
BENCHMARK_CORPUS.mdspells this out.Generated by Claude Code