Skip to content

Define the large-asset benchmark corpus - #47

Merged
snkmcb merged 4 commits into
mainfrom
claude/adoring-babbage-lf9ws8
Nov 3, 2026
Merged

snkmcb merged 4 commits into
mainfrom
claude/adoring-babbage-lf9ws8

Conversation

@piggy-penguin

Copy link
Copy Markdown
Member

Summary

Implements the next v0.6.0 immediate action in docs/roadmap/current.md: define the large-asset benchmark corpus.

  • tests/benchmark/corpus.json declares four tiers, each with a recorded license:

    • fixture: the committed synthetic fixtures, one per format.
    • corpus: the six committed CC0 8,192-Gaussian subsets.
    • scale: a generated ladder of 1,048,576 / 4,194,304 / 7,995,392 Gaussians in PLY, SPZ, and SOG. The top step is the largest whole number of tiles under the 8M import ceiling.
    • real: full-size exports kept outside the repository. These are the four retained CC0 source exports, the local Postshot cactus files, and the Graphdeco garden/bicycle, which are download-on-demand and non-commercial.

    The manifest also sets provisional resource targets: metadata-only read and CanRead at most 0.25 s, and peak resident memory at most 0.35 GiB + 0.60 GiB per million Gaussians.

  • scripts/benchmark_corpus.py has four commands:

    • list and verify.
    • generate: tiles the committed CC0 yashica-t4-top8192.ply on a grid and encodes all three formats from the same rows. It checks the output against the recorded hashes: file SHA-256 for PLY and SOG, and the decompressed-payload SHA-256 for SPZ.
    • run: measures each asset through benchmark_import.py, one process per asset, and gates the result on the expected count, SH degree, and targets.
  • The SOG fixture generator gains webp_lossless_planar. It produces the same VP8L bitstream as the reference encoder, built from per-channel bytes so it works on multi-million-texel planes. The committed fixtures regenerate byte-identical.

  • CTest, composed root build only: benchmark_corpus_manifest runs verify, and benchmark_corpus_ci measures the fixture, corpus, and 1M rows. Both run in the workspace-ctest-pr-* cells.

  • Fix: on Linux, benchmark_import.py now reads peak memory from VmHWM. ru_maxrss starts from the parent's RSS at fork and survives exec, so it inflated the peak when a large parent launched the benchmark.

  • Docs: new docs/reference/BENCHMARK_CORPUS.md. The docs index, BUILDING.md, WORKSPACE.md, PERFORMANCE_BASELINES.md, the roadmap, the backlog, and the CHANGELOG are updated.

Why?

The roadmap's step 2 is to "define and measure large-asset validation targets". No redistributable real asset reaches millions of Gaussians, so this adds a deterministic generated ladder up to the import ceiling. It sits beside the real references that design policy §12.1 names.

Validation

  • python scripts/benchmark_corpus.py verify: 26 assets, 0 problems. The planar encoder matches the reference encoder at four image sizes.
  • generate --tier scale: all nine assets generated in about 2 minutes. A second generate --ci into a fresh directory reproduced the recorded hashes.
  • The nine scale assets were decoded locally on Linux with the three bundles' C++ decoders, outside USD. All decode with 0 CheckCloudContract violations, and decoded positions agree across formats within quantization.
  • plugins/gaussian-sog/tools/generate_fixtures.py regenerates the fixtures with no diff.
  • The run plumbing (gating, report, temp-dir cleanup) was exercised against a stub pxr. OpenUSD is not available in this environment, so the real benchmark_corpus_ci measurement is left to the hosted workspace CTest cells on this PR.

Checklist

  • I linked related context or explained why there is no issue.
  • I ran the relevant tests and checks, or described what remains.
  • I updated documentation or tests when behavior changed.
  • I did not add private or unlicensed assets. Generated assets are derived from a committed CC0 subset and are never committed.

Reviewer notes

  • The targets are provisional until hosted rows exist. If a cell fails a memory or time bound, that is a measurement to record first, not a reason to loosen the bound silently.
  • The scale tier represents costs that grow with Gaussian count. It does not represent costs that grow with source bytes: SPZ repeats deflate well, and SOG planes use the uncompressed flat-code VP8L. BENCHMARK_CORPUS.md spells this out.

Generated by Claude Code

ru_maxrss starts from the parent's resident size at fork and survives
exec, so a benchmark launched by a large process reported the parent's
memory as part of its own peak. VmHWM is the process image's own
high-water mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
tests/benchmark/corpus.json declares four tiers: the committed fixtures,
the committed CC0 subsets, a generated scale ladder (1,048,576,
4,194,304, and 7,995,392 Gaussians) in PLY, SPZ, and SOG, and real
exports kept outside the repository, plus provisional resource targets.

scripts/benchmark_corpus.py generates the ladder deterministically by
tiling the committed CC0 yashica-t4-top8192.ply on a grid, encodes all
three formats from the same rows, checks the output against recorded
hashes, and measures any tier through benchmark_import.py one process
per asset. The SOG fixture generator gains a planar VP8L encoder that
emits the reference encoder's bytes for multi-million-texel planes; the
committed fixtures regenerate unchanged.

The composed root build registers benchmark_corpus_manifest (manifest,
committed hashes, encoder equivalence) and benchmark_corpus_ci (fixture,
corpus, and 1M rows against the targets), so the workspace CTest cells
run both on every hosted platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JQZ6zA7Tyd5itEG89Gj5hR
@snkmcb
snkmcb marked this pull request as ready for review November 3, 2026 10:20
@snkmcb
snkmcb merged commit f5edcba into main Nov 3, 2026
13 checks passed
@snkmcb
snkmcb deleted the claude/adoring-babbage-lf9ws8 branch November 3, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants