Holo4D: Geometry-Conditioned Novel-View Video Generation

Yushi Lan

Holo4D generates a target-camera video from one dynamic source video, persistent 3D trajectories, RGB-D geometry, and camera rays. This release contains the complete 384p training, inference, long-rollout, and evaluation pipeline used by the released checkpoints.

Highlights

• 14B Wan 2.1 V2V generation at 384 × 672 × 49 frames.
• Real-video reshooting with VGGT-Omega + CoWTracker geometry.
• Native I2V continuation with an arbitrary number of 49-frame chunks.
• Generated-boundary handoff: no ground-truth target image enters a continuation chunk.
• Training configs, metric suites, deterministic smoke assets, and public checkpoints are included.

Installation

Python 3.12 is the tested environment. Install a CUDA-matched PyTorch build first, then Holo4D:

conda create -n holo4d python=3.12 -y
conda activate holo4d
pip install -r requirements/cu130.txt
pip install -e .

The launchers default to FlashAttention 2. Set HOLO4D_FLASH_ATTN3=1 only in an environment containing exactly one compatible FlashAttention 3 implementation.

Models and data

Public inference launchers automatically download and cache the required Holo4D checkpoint and Wan 2.1 base through huggingface_hub. Wan downloads are pinned to the revisions whose T5/VAE objects match the CIVO runtime. The first run is large; later runs reuse ~/.cache/holo4d. Set HOLO4D_CACHE_DIR to move that cache, or provide local path variables to avoid downloads.

Hugging Face layout:

• checkpoints/t2v_49_chunk.safetensors: ordinary 49-frame generation and the first rollout chunk.
• checkpoints/i2v_49_rollout.safetensors: every generated-boundary continuation chunk.
• evaluation/syn4d_flying_group_2chunk_demo.npz: compact stride-1 384p input for both no-data-path demos.
• evaluation/vista4d_50_inputs.npz: 50 sources, two official target trajectories each, and matching VGGT-Omega/CoWTracker caches.
• evaluation/vista4d_50_reference_outputs.tar: 100 corrected canonical reference predictions.
• evaluation/public_code_validation_20260812.tar: public-code validation outputs and rollout boundary audits.

The bundled Vista4D material comes from Vista4D-Eval-Data and retains its upstream Apache-2.0 license and provenance.

Inference

One-command 49-frame and I2V demos

These commands require a CUDA GPU but no dataset path. On first use they download and cache the compact demo input, pinned Wan bases, and released Holo4D checkpoints:

# First 49-frame chunk
bash inference/demo_t2v_49.sh

# T2V first chunk + generated-boundary I2V continuation = 49 + 48 frames
bash inference/demo_i2v_97.sh

Checked reference outputs are included in evaluation/demo_outputs/: t2v_49_chunk.mp4 and i2v_49_plus_48_rollout.mp4.

Vista4D real-video reshooting

The following command downloads all missing public assets and generates the two official trajectories for avocado-slice:

bash inference/demo_vista4d.sh

Set VIDEO_NAMES to choose another source. Run bash inference/sample_vista4d.sh directly with no VIDEO_NAMES to generate all 50 sources (100 outputs).

The canonical Vista4D settings are 50 denoising steps, CFG 5, sigma shift 5, source-frame-0 trajectory anchoring, camera damping 0.5, Pi3-to-VGGT static-depth scale alignment, and focal-ratio transfer.

Syn4D 49-frame V2V

SYN4D_ROOT=/data/Syn4D_v1_stride_5 \
SYN4D_CAPTION_DIR=/data/Syn4D_caption_cache \
SEQUENCE_IDXS=0 CAMERA_PAIRS=0-1 \
OUTPUT_DIR=outputs/syn4d_49f \
bash inference/sample_19500_syn4d.sh

Long I2V rollout

SYN4D_ROOT=/data/Syn4D_v1_stride_5 \
SYN4D_CAPTION_DIR=/data/Syn4D_caption_cache \
FIXED_SOURCE_SEQUENCES=seq_000000 \
NUM_PAIRS=1 LONG_HORIZON_FRAMES=97 \
bash inference/sample_mixed_t2v_i2v_syn4d.sh

Set LONG_HORIZON_FRAMES to the desired total, for example LONG_HORIZON_FRAMES=97 or 193. Full chunks follow 49 + 48k; a partial final chunk is trimmed to the exact requested length. The launcher automatically creates the stitched video and provenance manifest.

Adjacent chunks share one boundary frame. Holo4D removes the repeated first frame of each continuation, so k chunks contain 49 + 48(k-1) unique frames.

Verified release outputs

The public launchers were rerun on August 13, 2026: ten Vista4D outputs from five sources decoded to exactly 49 frames at 672 × 384, and two Syn4D flying_group rollouts decoded to exactly 97 frames. For both rollouts, the saved seed endpoint and the image read by the I2V continuation had identical SHA-256 hashes.

The public Vista4D driver, camera construction, source-cache loader, and reprojection path match the final CIVO pipeline. A strict sequential comparison on the same H200, in the same Python/CUDA/FlashAttention environment, produced exactly equal BF16 latent tensors (max_abs = mean_abs = 0) and byte-identical MP4 files. See evaluation/SAME_H200_PARITY.md for the controlled comparison. Diffusion outputs are not promised to remain bitwise-identical across different GPU or software stacks; historical archived references are therefore qualitative references rather than golden byte fixtures.

The two one-command demo outputs are in evaluation/demo_outputs/, and ten Vista4D examples are in assets/vista4d/; both folders include checksummed manifests. The complete public-code validation bundle, including both long rollouts and boundary-condition audit files, is available from Hugging Face:

hf download yslan/Holo4D-2026 evaluation/public_code_validation_20260812.tar \
  --local-dir outputs/validation

Training

V2V model

train/train_384p.sh matches the released inference contract: V2V width concatenation, 49 frames at 384 × 672, persistent RGB-D trajectories, per-block injection, query-frame impulse, Plücker camera control, ray-map tokens, and sigma shift 5.

Set the data roots referenced by config/holo4d/train_384p.yaml:

export SYN4D_ROOT=/data/Syn4D
export SYN4D_CAPTION_DIR=/data/captions/Syn4D
export RECAMMASTER_ROOT=/data/MultiCamVideo-Dataset
export RECAMMASTER_CAPTION_CACHE_DIR=/data/captions/recammaster
export MULTICAM_FG_MASK_ROOT=/data/recammaster_masks
export RECAMMASTER_SOURCE49_ROOT=/data/recammaster_vggt
export RECAMMASTER_PAIR1X49_LABEL_ROOT=/data/recammaster_scale_labels
export RECAMMASTER_COW_ROOT=/data/recammaster_cow
export DL3DV_ROOT=/data/dl3dv
export MVS_SYNTH_ROOT=/data/mvs_synth

OUTPUT_PATH=outputs/train_384p \
NUM_PROCESSES=8 \
bash train/train_384p.sh

Use RESUME_CHECKPOINT=/path/to/step-N.safetensors to continue full-parameter training. The launcher exposes learning rate, epoch length, checkpoint interval, workers, gradient accumulation, and condition dropout through environment variables.

I2V continuation model

The released continuation model uses the Wan 2.1 I2V 36-channel base, CLIP image encoder, one frame-0 trajectory query, target-boundary dropout 0.3, and image-latent noise 0.05. In addition to the roots above, set the OmniDROID inputs declared by config/holo4d/train_i2v_384p.yaml:

export OMNI_DROID_ROOT=/data/omnidroid
export OMNI_DROID_INDEX=/data/omnidroid/index.json
export OMNI_DROID_CACHE_MANIFEST=/data/omnidroid/cache_manifest.json
export DROID_NVS_INDEX=/data/droid/nvs_index.json
export OMNI_CAPTION_DIR=/data/captions/omnidroid

RESUME_CHECKPOINT=/path/to/an/i2v-training-checkpoint.safetensors \
OUTPUT_PATH=outputs/train_i2v_384p \
NUM_PROCESSES=8 \
bash train/train_i2v_384p.sh

Omit RESUME_CHECKPOINT to initialize from the Wan I2V base; provide it to continue a Holo4D I2V run. The V2V and I2V launchers share the same geometry/camera contract, trainer, resolution, and noise schedule.

Evaluation

The evaluation suite is under eval/ and covers video quality, camera accuracy, Syn4D, DAVIS, and iPhone/DyCheck protocols.

# Standardized video metrics for an output tree
METHOD=holo4d METHOD_PRED_ROOT=/path/to/predictions \
DAVIS_ROOT=/path/to/DAVIS/JPEGImages/480p \
PAIRS_FULL_CSV=/path/to/davis_trajectories_full.csv \
EVAL_ROOT=outputs/eval bash eval/common/eval_video_metrics.sh

# VGGT-Omega camera-trajectory metrics
METHOD=holo4d METHOD_PRED_ROOT=/path/to/predictions \
PAIRS_CSV=/path/to/davis_trajectories.csv CAMERAS_ROOT=/path/to/cameras_2traj \
VGGT_OMEGA_REPO=/path/to/vggt-omega \
VGGT_OMEGA_CHECKPOINT=/path/to/vggt_omega_1b_512.pt \
EVAL_ROOT=outputs/eval bash eval/common/eval_camera_metrics_vggt_omega.sh

# Lightweight repository tests
pip install -e ".[dev]"
bash scripts/test_release.sh

The video command defaults to clip_t,clip_f,clip_v. Set METRICS=fvd,clip_t,clip_f,clip_v after placing the two I3D weights named by the launcher under eval/common/video_quality/common_metrics_on_video_quality/fvd/.

See eval/README.md for dataset layouts and metric-specific commands, evaluation/SAME_H200_PARITY.md for the strict original/public same-stack parity record, and evaluation/A100_REPRODUCTION.md for the public-code reproduction record.

Code structure

Holo4D/
├── holo4d/                  # geometry, trajectories, data, cache, and rollout logic
├── holo4d_runtime/          # vendored Wan diffusion/model runtime
├── examples/wan2.1_fun/     # canonical Python inference entrypoints
├── inference/               # ready-to-run 49f, Vista4D, and long-rollout launchers
├── train/                   # 384p trainer and launcher
├── config/holo4d/           # model and production data recipes
├── eval/                    # video, camera, and dataset-specific metrics
├── assets/                  # smoke inputs and qualitative examples
└── requirements/            # tested CUDA dependency profiles

Reproducibility notes

Keep checkpoint, Wan base, seed, resolution, denoising steps, CFG, sigma shift, geometry backend, crop, and trajectory settings fixed when comparing outputs. Each long-rollout case records both checkpoints and its generated boundary image; incompatible existing artifacts are not silently reused.

Citation

@software{lan2026holo4d,
  author = {Yushi Lan},
  title = {Holo4D: Geometry-Conditioned Novel-View Video Generation},
  year = {2026},
  url = {https://github.com/NIRVANALAN/Holo4D}
}

Acknowledgements

Holo4D's Wan 2.1 runtime and generation pipeline were developed from DiffSynth-Studio. Users who prefer to obtain or configure the Wan 2.1 base weights manually should follow the model instructions in that repository.

Holo4D is released under the Apache License 2.0. The Wan 2.1 models and external datasets remain subject to their respective licenses.