LastKV-NVS-Objaverse

LastKV for object-centric novel-view synthesis, trained on Objaverse with a three-stage progressive-resolution curriculum: 256 × 256 → 512 × 512 → 1024 × 1024. This repository releases the third-stage checkpoint at 1024 × 1024.

A 24-layer Transformer with hidden width 768, patch size 8, and 184.8M parameters. LastKV shares the final layer's historical KV cache across attention layers. Training uses BF16 autocast; the released model weights are stored in FP32.

Training curriculum

Stage 1 starts from random initialization. At each resolution transition, only model weights are inherited; the optimizer, learning-rate scheduler, and step counters are reset.

Checkpoint steps are stage-local forward/backward pass counters (fwdbwd_pass_step); successful optimizer-update counts can be lower when updates are skipped. The selected checkpoints define the ancestry of the released weights. The configured budgets determine the cosine schedule and should be preserved when reproducing this training recipe, even when an earlier checkpoint is selected.

Training settings

• Optimizer: AdamW, betas (0.9, 0.95), weight decay 0.05.
• Schedule: 2,500-step learning-rate warmup followed by cosine decay in each stage.
• Supervision: pixel L2 plus 0.5 × VGG19 PerceptualLoss.
• Loss warmup: the first 4,000 passes of stage 1 use only L2. Stages 2 and 3 enable the perceptual term from the start.
• Training: 12 views per sequence, with teacher-forced sequential conditioning.

The VGG19 training perceptual loss and the LPIPS evaluation metrics use different definitions.

Checkpoint and use

Download model.pt. This is a PyTorch training checkpoint containing model, optimizer, lr_scheduler, fwdbwd_pass_step, and param_update_step.

The released checkpoint contains 6,980 stage-3 passes and 6,972 successful parameter updates. Its optimizer and scheduler states belong to stage 3.

To download and extract the model weights:

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("ZefanCai/LastKV-NVS-Objaverse", "model.pt")
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
state_dict = checkpoint["model"]

Rendering requires the matching LastKV NVS implementation, the 1024 × 1024 configuration, and compatible camera preprocessing. See checkpoint_info.json for the file size, SHA-256, and tensor metadata.

Evaluation of the released checkpoint

Evaluation uses GSO, with 1,018 objects, 1024 × 1024 images, 24 fixed views per object, and seed 4242. Sequential conditioning uses ground-truth preceding views. Metrics are averaged over the 23 target views within each object, then across objects.

Project page · Paper