Measured on device (edge-compat): browser · Chromium 151 on M4 Max · LiteRT.js 2.5.3 · WebGPU · 285 ms p50 · output differs from CPU (max rel diff 2.3e+10) (2026-08-20). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/tipsv2-b14-dpt/CARD.md
TIPSv2 (Google DeepMind, CVPR 2026)
is a ViT-B/14 vision-language backbone. This repository provides LiteRT exports
of google/tipsv2-b14-dpt, with one
shared backbone and three DPT heads producing metric depth, surface normals,
and ADE20K semantic segmentation from a single 448 × 448 image.
The model has 12 transformer blocks, width 768, 12 attention heads, and one register token. The DPT heads use taps at blocks 3, 6, 9, and 12: approximately 86 million backbone parameters and 72 million head parameters. For image embeddings without DPT heads, see the TIPSv2-B14 image encoder.
Example FP16 GPU output on Pixel 8a. Left to right: input, metric depth, surface normals, and segmentation. Hero photo: Pexels.
| File | Format | Size (MiB) | Notes |
|---|---|---|---|
| tipsv2_b14_dpt_fp16.tflite | FP16 weights, tanh GELU | 303.6 | GPU compatibility forms; tested on Mali and Adreno |
| tipsv2_b14_dpt_erf_fp16.tflite | FP16 weights, exact GELU | 303.5 | Builtin GELU; tested on Adreno and Qualcomm NPU |
| tipsv2_b14_dpt_a16w8.tflite | A16W8 | 159.6 | Static activation quantization; tested with MediaTek NPU compilation |
| tipsv2_b14_dpt_a16w8_qc.tflite | A16W8 compilation input | 159.6 | Activation storage and LayerNorm signature adapted for Qualcomm NPU |
A16W8 uses INT8 weights and calibrated 16-bit activation regions. The _qc
file preserves the calibrated real values using unsigned activation storage;
it requires NPU compilation and does not support LiteRT CPU fallback. NPU use
requires the appropriate vendor compiler and runtime. File size describes
storage, not peak runtime memory or inference speed. FP16 describes weight
storage; the accelerator compiler chooses the internal arithmetic.
All files use signature serving_default, with FLOAT32 public tensors:
| Signature name | Shape | Meaning |
|---|---|---|
args_0 |
[1, 3, 448, 448] |
RGB input, NCHW, values in [0, 1] |
output_0 |
[1, 1, 448, 448] |
Metric depth in meters |
output_1 |
[1, 3, 448, 448] |
Surface-normal vectors |
output_2 |
[1, 150, 256, 256] |
ADE20K segmentation logits |
Convert to RGB, resize directly to 448 × 448 using PIL bilinear interpolation, divide by 255, and transpose to NCHW. There is no center crop or ImageNet mean/std normalization. When changing output resolution, resize segmentation logits before taking argmax and renormalize normal vectors after resizing.
output_2 holds the segmentation logits at the DPT head's native grid —
argmax over the 150 channels host-side, then upscale (the official pipeline bilinearly
upsamples the logits to 448 first; argmax agreement between the two is 99.96 %).
Compile the selected TFLite file for the target SoC using LiteRT's vendor
compiler plugin, then load the compiled artifact with Accelerator.NPU in the
Android example below. Use the _qc file for the tested Qualcomm A16W8 path.
The files in this repository are compiler inputs, not bundled vendor binaries.
The A16W8 Qualcomm path was tested with public LiteRT 2.3.0 and QAIRT 2.50. MediaTek measurements use NeuroPilot 8 and a locally patched LiteRT source build; Qualcomm FP16 measurements use that source build with QAIRT 2.50. These source-build results do not establish compatibility with unmodified released compiler packages. Recheck compatibility when changing the SDK or runtime. Runtime setup examples are available in the NPU recipe and GPU recipe.
The existing C++ run_model tool can run a compiled model using an input
directory containing args_0.raw:
run_model --graph=model_aot.tflite --accelerator=npu \
--dispatch_library_dir=/path/to/runtime --input_dir=/path/to/inputs
// 318 MB — stage into filesDir (adb push + run-as cp) rather than APK assets
val model = CompiledModel.create(File(context.filesDir, "tipsv2_b14_dpt_fp16.tflite").absolutePath,
CompiledModel.Options(Accelerator.GPU), null)
val inputs = model.createInputBuffers()
val outputs = model.createOutputBuffers()
inputs[0].writeFloat(imageNchw01) // [1,3,448,448] RGB / 255, no mean/std
model.run(inputs, outputs)
val depth = outputs[0].readFloat() // [448*448] metres
val normals = outputs[1].readFloat() // [3*448*448] unit vectors
val seg = outputs[2].readFloat() // [150*256*256] logits -> argmax per pixel host-side
import numpy as np
from ai_edge_litert.compiled_model import CompiledModel
model = CompiledModel.from_file("tipsv2_b14_dpt_fp16.tflite")
inputs = model.create_input_buffers(0)
outputs = model.create_output_buffers(0)
inputs[0].write(np.ascontiguousarray(image, np.float32)) # [1,3,448,448] in [0,1]
model.run_by_index(0, inputs, outputs)
depth = outputs[0].read(448 * 448, np.float32).reshape(448, 448) # metres
normals = outputs[1].read(3 * 448 * 448, np.float32).reshape(3, 448, 448)
seg = outputs[2].read(150 * 256 * 256, np.float32).reshape(150, 256, 256)
labels = seg.argmax(0) # ADE20K ids (256x256)
The TIPSv2 Android sample (Kotlin, CompiledModel GPU): photo picker → input | depth,
normals | segmentation with an ADE20K legend. The conversion script build_tipsv2.py
(re-authoring + parity + litert-torch convert + fp16 + device harness) is included in this repo.
Measured by @snnn001.
The paired evaluation uses 200 NYUv2 validation images and 500 ADE20K validation images, selected uniformly without replacement with seed 2026. These subset results use the protocol below; they are not reproductions of the paper's full benchmark.
(0.001, 10]; predictions are resized bilinearly and clipped to [0.001, 10].
Metrics are averaged per image.| Execution / format | Depth AbsRel ↓ | Depth RMSE (m) ↓ | Depth δ1 ↑ | ADE20K mIoU ↑ |
|---|---|---|---|---|
| PyTorch FP32 checkpoint | 0.1034 | 0.388 | 91.16% | 43.20% |
SM-S937U1 NPU / FP16, tanh GELU |
0.1034 | 0.388 | 91.13% | 43.21% |
SM-S937U1 NPU / FP16, exact GELU |
0.1038 | 0.388 | 91.17% | 43.21% |
SM-S937U1 NPU / A16W8 |
0.1038 | 0.389 | 90.99% | 43.23% |
TECNO-LJ9 NPU / A16W8 |
0.1038 | 0.389 | 90.99% | 43.22% |
Mean normal-vector differences from the PyTorch FP32 reference on the same 200 images are 0.10° for FP16/tanh GELU, 0.42° for FP16/exact GELU, and 0.84° for A16W8 on SM-S937U1; A16W8 on TECNO-LJ9 gives 0.83°. These measure numerical agreement, not ground-truth normal accuracy; the dataset's normal coordinate convention has not been independently established.
All reported NPU accuracy runs checked every output for finite values. The MediaTek FP16 attempts did not produce predictions because of the runtime allocation failure described below.
These are representative measurements to give a rough sense of runtime
behavior, not a direct comparison between hardware platforms. AOT results
were collected with LiteRT's run_model launched through adb, rather than an
integrated Android application. Timings are rounded to integer milliseconds.
| Device / SoC | Execution | Floating weight format | tanh GELU (ms) | exact GELU (ms) | A16W8 (ms) | Measured by |
|---|---|---|---|---|---|---|
SM-S937U1 / SM8750 |
NPU AOT | FP16 | 377 / 391 | 202 / 230 | 222 / 231 | @snnn001 |
TECNO-LJ9 / MT6897 |
NPU AOT | FP16 | Allocation failure | Allocation failure | 546 / 589 | @snnn001 |
| SM8850 | NPU JIT | FP32 | 327 | 142 | 129 | @mlboydaisuke (FP32); @snnn001 (A16W8, SM-S942U1) |
| SM8850 | Adreno GPU | FP32 | 283 | 273 | — | @mlboydaisuke |
V2610 |
NPU JIT | FP16 | 272 | 279 | 185 | @snnn001 |
V2610 |
GPU | FP16 | 299 | 343 | — | @snnn001 |
| Pixel 8a / Tensor G3 | Mali GPU | FP16 | ~920 | — | — | @mlboydaisuke |
| Raspberry Pi 5 | CPU, 4 threads | FP16 | 11,185 | 10,643 | — | @mlboydaisuke |
For AOT, entries show p50 / p95, each the median across three five-minute
trials, with five warmups and a one-minute cooldown before each trial. Timings
measure synchronous model invocation and exclude preprocessing and input/output
copies. Successful AOT rows execute in one NPU partition with no CPU fallback.
The A16W8 column uses _qc on SM-S937U1 and the standard file on TECNO-LJ9.
The sustained SM-S937U1 trials reached Android thermal status 2 (moderate)
for FP16/tanh GELU and A16W8, and 3 (severe) for FP16/exact GELU. Exact-GELU
trial medians ranged from 198 to 223 ms. These timings include the device's
thermal management. Android thermal status definitions.
Allocation failure: on the tested MT6897 AOT setup, both FP16 files produce eight NPU partitions with seven FC operations and one ADD on CPU. Execution fails while allocating an Android hardware buffer before the first prediction; there is no valid latency or accuracy result for that setup.
The SM8850 measurements use 50 timed runs after warmup, one accelerator per
process, and thermal status NONE. The floating-point columns were reported
with LiteRT 2.2.0 for FP32 builds of the matching tanh/exact-GELU graphs, not
the downloadable FP16-weight files (on SM-S942U1 / SM8850, the downloadable
FP16-weight files run at 189 ms for tanh GELU and 130 ms for exact GELU on NPU
JIT, and 244 ms / 242 ms on Adreno GPU). The SM8850 A16W8 NPU JIT result
(129 ms median, p50 / p95 of 129 / 131 ms; 130 / 131 ms with NPU AOT) was
measured on a Samsung Galaxy S26 (SM-S942U1) using
tipsv2_b14_dpt_a16w8_qc.tflite, which delegates all 775 operations to a
single Qualcomm HTP partition (first-load JIT compile ~41 s, cached load
~310–430 ms). A16W8 is unsupported by the LITERT_CL OpenCL GPU delegate on
Adreno GPU because 16-bit integer activation tensors (INT16 / UINT16) are
not supported by the GPU kernels.
The Pixel 8a GPU result uses CompiledModel/LITERT_CL, with all 1,434
nodes in one GPU partition; latency is a wall-time estimate from one versus
six consecutive invocations. Initial GPU compile/load is approximately five
seconds. The Raspberry Pi 5 CPU results use XNNPACK with four threads,
10 warmups and up to 50 timed runs per invocation, repeated three times with
a 150-second phase cap; 42 tanh and 45 exact-GELU runs completed. No throttling
was reported. These protocols differ from the AOT trials.
Hardware benchmark disclosure for the AOT measurements: Results were measured by @snnn001 on retail devices purchased through normal channels. These results are not affiliated with, sponsored by, endorsed by, or verified by Samsung, vivo, TECNO, Qualcomm, MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU, OS build, thermal state, battery mode, backend, model quantization, runtime version, and benchmark settings.
A WebGPU compatibility report records a numerical mismatch for its tested browser configuration; its timing should not be treated as a validated inference result.
The NPU is faster — once the GELU is written as an op. The original file loses on
the NPU (326.9 ms vs 282.5 ms on the GPU), and the cause is the tanh-GELU elementwise
decomposition (24 sites: ViT blocks + DPT readout), which falls off the Hexagon
compiler's fast path. tipsv2_b14_dpt_erf_fp16.tflite is the same model with the
GELU emitted as the builtin GELU op (exact erf); all three outputs match the original
file at corr 0.99998+ (depth 0.999984, normals 0.999999, seg 0.999993):
| file | NPU (Hexagon v81) | GPU (Adreno) |
|---|---|---|
tipsv2_b14_dpt_fp16.tflite (tanh-GELU) |
326.9 ms | 282.5 ms |
tipsv2_b14_dpt_erf_fp16.tflite (GELU op) |
142.0 ms | 273.3 ms |
Pick by accelerator: the erf file for the NPU (2.30x). It also ran at full speed on
this Adreno GPU; the Mali path is unverified with the builtin GELU op — keep the
original file for Mali. The erf build is build_tipsv2.py with the gelu function
set to torch.nn.functional.gelu.
Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81,
Android 16), LiteRT CompiledModel 2.2.0, on-device JIT compile (first load
compiles in ~2-4 min and caches; later loads ~0.4-0.8 s), one accelerator per process,
warm-up then N=50 timed runs, median reported, every row at thermal status NONE —
which also settles the direction this section previously called provisional.
Latencies were taken on the fp32 build of the same graph; fp16 weight storage measured
within noise on the models where the fold test was run (DINOv2, zipformer).
The runtime libraries the NPU needs are in the NPU recipe; GPU wiring is in the GPU recipe.
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).
| File | Inference (median) | Spread (min–max) | Runs | Peak memory |
|---|---|---|---|---|
tipsv2_b14_dpt_erf_fp16.tflite |
10,642.7 ms | 10,582.8–10,762.4 ms | 45 | 1291 MB |
tipsv2_b14_dpt_fp16.tflite |
11,185.4 ms | 11,096.1–11,293.2 ms | 42 | 1292 MB |
build_tipsv2.py provides the FP16 conversion, parity checks,
and device harness using litert-torch. GPU compatibility rewrites include
four-dimensional attention, folded LayerScale, range-safe LayerNorm, split
CLS/patch readout, and equivalents for transpose convolutions and bilinear
upsampling. The two FP16 files differ in their GELU representation.
The A16W8 recipe uses TorchAO 0.18.0 static post-training quantization with 500 training images: 250 NYUv2 and 250 ADE20K, separate from evaluation, with seed 2026. Fold LayerScale before calibration; use min/max calibration, symmetric INT8 weights per output channel, symmetric INT16 activations per tensor, and INT64 biases for learned FC and ordinary convolution layers. Eligible MLP, attention, residual, and LayerNorm regions retain integer interfaces; transpose convolutions and unsupported surrounding operations remain floating point.
The A16W8 depth branch uses power-of-two rescaling to control intermediate
magnitudes and a positive normalization floor of 2^-14. This floor changes
the regularizer; the reported comparison against PyTorch includes that effect.
The source is google/tipsv2-b14-dpt.
Recreating the TFLite files also requires export support for the integer
operators and normalization composites.
Fully GPU-resident on a Pixel 8a (1434/1434 nodes, 1 partition, ~0.9 s for all three heads). Re-authored from the HF weights with exact rewrites:
[1, heads, N, d] with a manual
softmax(qkᵀ/√d)·v — the delegate rejects the native 5D head-split.attn.proj / mlp.fc2; SafeLayerNorm (deviation pre-scaled by
1/64 before squaring so the fp16 variance cannot overflow); tanh-GELU (no ERF kernel —
the only non-exact rewrite, corr 0.999998 vs the official model).cat(patch, cls.expand) @ W → patch @ W_a + cls @ W_b (exact, no
BROADCAST_TO); ConvTranspose2d(k=s) → zero-stuff + Conv2d (exact; TRANSPOSE_CONV is
rejected on Pixel 8a); the fusion blocks' bilinear ×2 align_corners=True (banned on the GPU)
→ two constant-RHS matmuls U·X·Uᵀ (exact).relu(l)/Σrelu(l)), so
power-of-2 scales are folded into its weights/biases (per-level convs 1/4096…1/64, each fusion
out_conv ×1/4, project 1/32, head 1/16 — matching scales at every residual add) keeping
every stage ≲100. Bit-exact in fp32.Parity (desktop fp32 re-authored vs official PyTorch): depth corr 0.999998 · normals 0.999999 ·
seg argmax 99.96 %. Device fp16 (Pixel 8a GPU) vs desktop fp32: depth corr 0.99986 · normals
0.99990 · seg argmax 99.3 %. Conversion script: build_tipsv2.py (litert-torch).
Apache-2.0 (TIPSv2 / Google DeepMind). Hero photo: Pexels (free license).
google/tipsv2-b14-dpt