See our collection for all versions of TIPSv2-DPT.

Run TIPSv2-DPT with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/tipsv2-so400m14-dpt

Paper: TIPSv2 (arXiv:2604.12012)

TIPSv2-DPT stacks DPT (Dense Prediction Transformer) heads on the TIPSv2 vision backbone. This single checkpoint serves three task classes: monocular depth estimation and semantic segmentation, or both at once.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of google/tipsv2-so400m14-dpt for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
import numpy as np
import keras
from zeromodels.models.tipsv2_dpt import (
    Tipsv2DptDensePredict,        # depth + segmentation
    Tipsv2DptDepthEstimation,     # depth only
    Tipsv2DptSemanticSegment,     # segmentation only
    Tipsv2DptImageProcessor,
)

# all three load from the SAME repo
model = Tipsv2DptDensePredict.from_weights("zeromodels/tipsv2-so400m14-dpt")
proc = Tipsv2DptImageProcessor(image_resolution=448)

image = Image.open("your_image.jpg").convert("RGB")
pixel_values = proc(np.array(image))["pixel_values"]
out = model(pixel_values)
depth = keras.ops.convert_to_numpy(out["predicted_depth"])          # (1, H', W')
seg = keras.ops.convert_to_numpy(out["segmentation_logits"])        # (1, H', W', num_labels)

# single-task variants (same weights, one output each)
depth_model = Tipsv2DptDepthEstimation.from_weights("zeromodels/tipsv2-so400m14-dpt")
seg_model = Tipsv2DptSemanticSegment.from_weights("zeromodels/tipsv2-so400m14-dpt")

Variants:

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• The image processor rescales to [0, 1] (no mean/std normalization); input resolution is 448.
• Outputs are at the DPT feature resolution; resize to the input size for visualization.
• Upstream checkpoint: Tipsv2DptDensePredict.from_weights("hf:google/tipsv2-so400m14-dpt").

Special Thanks

A huge thank you to the TIPSv2 authors (Google DeepMind) and the HF community.

License: Apache-2.0 (matches the upstream google/tipsv2-so400m14-dpt checkpoint).