LEGO: Leveled Language Gaussian Splatting

Official implementation of LEGO: Leveled Language Gaussian Splatting.

Yuning Peng, Haiping Wang, Yuan Liu, Yipeng Lu, Zhen Dong, and Bisheng Yang
ECCV 2026
Project Page | Paper

📖 Overview

TL;DR: LEGO turns perspective-dependent, multi-granular 2D masks into a unified leveled 3D Gaussian field for multi-granular open-vocabulary scene understanding.

Foundation models such as SAM expose rich object structures in individual images, but their masks are tied to viewpoint and lack cross-view granular consensus. Absolute physical scale is also an unreliable semantic cue: instances of one category can have very different sizes. LEGO instead represents granularity as an intrinsic structural level, independent of absolute object size.

Given multi-view images, LEGO first reconstructs scene geometry with MASt3R and uses spatial co-visibility and 3D scale to assign SAM masks to consistent coarse-to-fine levels. These masks supervise hierarchical contrastive distillation into a 2D Gaussian field. Recursive clustering produces a tree-structured scene decomposition; selected views then provide CLIP embeddings for direct open-vocabulary queries and relation-aware chain-of-reasoning (CoR) search.

🔧 Installation

The validated environment is Ubuntu 20.04 with Python 3.11, PyTorch 2.5.1, CUDA 12.1, and an NVIDIA GPU with 24GB VRAM. The setup script creates an isolated Conda environment and builds the included CUDA extensions:

bash scripts/setup_env.sh
conda activate lego

Model Weights

Download the four local checkpoints after reading the MASt3R checkpoint terms:

bash scripts/download_models.sh --accept-mast3r-license

The OpenCLIP weights are fetched from Hugging Face and cached on first use.

Optionally run lego doctor to verify the environment, compiled extensions, and model checkpoints.

📂 Data

Set the dataset and output roots, then place each source scene in its dataset directory. Symbolic links are supported. Training artifacts stay under the output root; the loader only adds a derived images_N/ directory beside images/ when the configured downsampling factor requires it.

export LEGO_DATA_ROOT=/path/to/datasets
export LEGO_OUTPUT_ROOT=/path/to/outputs

<dataset_root>
|---mipnerf360
|   |---room
|   |   |---images
|   |   |---sparse
|   |       |---0
|---lerf
|---...
|---cor
|   |---manifest.yaml
|   |---labels
|       |---teatime
|       |---kitchen
|       |---bonsai
|       |---counter

SPIn-NeRF configs map mixed-source scenes to their original LLFF, NeRF-Real-360, or Tanks and Temples directories.

The CoR benchmark contains 120 complex query cases (30 per scene) on 20 annotated views. Each query entails part-whole and spatial relationships. The teatime and kitchen cases use the LERF teatime and waldo_kitchen scenes; bonsai and counter use the corresponding Mip-NeRF 360 scenes.

To evaluate on the CoR benchmark, download the annotations from Google Drive. First place the four source scenes under lerf/ and mipnerf360/ as shown above, then place the extracted annotation directory at <dataset_root>/cor. The CoR scene configs resolve each logical benchmark scene to its source RGB/COLMAP directory automatically.

🔧 Training

LEGO processes each scene through geometry-aware mask leveling, RGB and hierarchical Gaussian training, then scene-tree construction and CLIP feature extraction. The complete workflow can run as one command or as a selected range of stages.

Inspect the configuration, then run the complete image-to-language-scene pipeline:

lego config 3d_ovs/room
lego run 3d_ovs/room
lego validate 3d_ovs/room

The runner executes mask generation, MASt3R reconstruction, mask leveling, RGB Gaussian training, hierarchical feature distillation, recursive clustering, relation-graph construction, view selection, SAM matching, and CLIP extraction. Each completed stage records its exact command and is reused while its command, expected outputs, and recorded upstream stages remain unchanged. Rerunning an upstream stage invalidates its downstream stage records. A failed or interrupted run can therefore be restarted with the same command.

Outputs for one scene stay together:

<output_root>
|---<dataset>
|   |---<scene>
|       |---resolved_config.yaml
|       |---masks
|       |   |---raw
|       |   |---mapping
|       |   |---leveled
|       |---reconstruction
|       |---ckpts
|       |---clustering
|           |---label_matrix.npz
|           |---cluster_tree.json
|           |---relation_graph.npz
|           |---cluster_features_CLIP.npz

To run only part of the pipeline or temporarily override a setting:

lego run 3d_ovs/room --from-stage train-rgb --to-stage build-tree
lego run 3d_ovs/room --set gaussian.init=sfm

Training configuration

• data.factor controls reconstruction and Gaussian training resolution. semantics.factor can override the resolution used for semantic feature extraction.
• gaussian.init=both retains all COLMAP points and fills the remaining init_num_points budget with sampled MASt3R points. Alternatives are sfm, mast3r, and random. Camera poses use COLMAP by default and can use MASt3R when gaussian.init=mast3r.
• gaussian.sample_type affects only MASt3R point sampling. Options are adaptive (default) and random.
• hierarchy.num_levels=L creates feature channels for scene levels 1 through L. Level 0 has no learned feature channel and denotes either an unassigned mask or the scene-tree root.
• clustering.end_level=K is inclusive: it clusters scene levels 1 through K. Public levels are always 1-based even though dense tensor columns are compact and zero-based internally.
• clustering.use_spatial appends normalized XYZ to HDBSCAN features. spatial_smooth and smooth_k control the first-level KNN label vote; UMAP and HDBSCAN parameters control recursive splitting.
• clustering.recursive_sample_limit=N deterministically fits recursive UMAP/HDBSCAN on at most N points per parent cluster and propagates the labels to all points. Set it to 0 for strict all-point clustering at substantially higher cost on oversized clusters.
• Temporary overrides use repeated --set key=value.

🖥️ Visualization

The Viser viewer loads the RGB field, learned hierarchy, clustered scene tree, and CLIP features from the scene output. Render modes include RGB, depth, normal, alpha, hierarchical features, clusters, and open-vocabulary query responses. CLIP features can directly retrieve scene objects at a selected hierarchy level.

viser-interaction-demo-web.mp4

lego viewer 3d_ovs/room --port 8080

Scene level 0 is the unpartitioned root. Levels 1 through L are the learned coarse-to-fine scene granularities shared by masks, checkpoints, clustering, evaluation, and viewer controls.

LLM-Guided Scene-Graph Queries

The scene graph separates two relation domains. intra_object stays within a subtree rooted at a level-1 object and resolves parts such as table -> leg. spatial can also cross node-contact edges between object subtrees, enabling chains such as table -> intra_object: leg -> spatial: ball.

LLM-guided queries use an OpenAI-compatible Chat Completions endpoint with JSON response-format support. The released configuration uses OpenRouter; set LLM_API_KEY in .env or the process environment and run:

lego viewer cor/teatime --scene-graph --port 8080

The LLM returns a typed anchor-to-target plan; CLIP scores and graph constraints execute the plan over the scene tree. To use another compatible provider, override reasoning.llm.base_url and reasoning.llm.model in the config. Cached evaluation plans are keyed by schema, prompt version, and model name.

📊 Evaluation

Evaluation renders the trained field and applies the protocol selected by the dataset config. LERF, Mip-NeRF 360, and 3D-OVS perform open-vocabulary localization and segmentation; NVOS and SPIn-NeRF propagate interactive annotations across views; COR evaluates relation-aware composed queries.

lego eval 3d_ovs/room

Every evaluator writes aggregate and per-example metrics, the resolved command, a complete log, and visualizations directly to:

<output_root>
|---<dataset>
|   |---eval
|       |---<scene>
|           |---eval.log
|           |---metrics.json
|           |---metrics.txt or metrics.csv
|           |---visualizations

Evaluation configuration

• evaluation.factor overrides the render resolution used by the evaluator without changing the trained field.
• evaluation.smoothing_kernel_ratio scales the smoothing kernel used by the shared LERF and Mip-NeRF 360 evaluator by the rendered image's short side.

Troubleshooting

• CUDA symbol or libcudart.so errors usually indicate an extension built under another PyTorch or CUDA version. Rebuild gsplat and MASt3R RoPE2D inside the active lego environment.
• The MASt3R retrieval codebook must remain beside the retrieval checkpoint with the released basename.
• Image stems must match generated mask stems after COLMAP sorting.
• Conda CUDA packages do not provide a kernel driver; verify the host driver with nvidia-smi.
• On offline machines, pre-populate the Hugging Face cache for the OpenCLIP model.

📄 License

This repository is released for non-commercial research under CC BY-NC-SA 4.0. MASt3R, DUSt3R, SAM, gsplat, OpenCLIP, and their checkpoints retain their original licenses and usage terms; see their upstream repositories and the license files under third_party/.

📇 Citation

@article{peng2026lego,
  title={LEGO: Leveled Language Gaussian Splatting},
  author={Peng, Yuning and Wang, Haiping and Liu, Yuan and Lu, Yipeng and Dong, Zhen and Yang, Bisheng},
  journal={arXiv preprint arXiv:2608.10057},
  year={2026}
}

🔗 Acknowledgements

This project builds on MASt3R, Segment Anything, OpenCLIP, and gsplat. We thank their authors for releasing their work.