API Try Docs Blog

d1-Omni-2B-ONNX

ONNX version of d1-Omni-2B for cross-platform inference with ONNX Runtime.

Find architecture and usage details in the original model card.

Files

---

Each precision pairs a decoder with a vision encoder and an audio encoder.

The fp16, q4f16 and q8f16 decoders run with fp16 activations, the others with fp32. q4 and q4f16 quantize only the MLP gate and up projections to int4 and keep the embedding, the LM head and the other layer projections in int8.

Use with ONNX Runtime

---

The vision encoder maps each image, or each temporal patch of a video, to projected features, and the audio encoder maps the log-mel features of 16 kHz audio to audio features. The decoder answers a state and up to 16 questions, written or spoken, packed into one row without padding, in a single forward pass with no cache. It takes input_ids, attention_mask, position_ids, the features at the <|image|>, <|video|> and <|audio|> placeholders (image_features, video_features, audio_features), question_ids (-1 on the state, k on question k) and the flat positions of the answer slots (slot_positions). It returns logits at the answer slots and the final-norm hidden_states.

The checkpoint's Python code in this repository builds the rows and turns these outputs into typed answers. For audio, image and video preprocessing, use the values in processor_config.json.

On GPU, use fp16, q4f16 or q8f16. The CUDA execution provider has no fp32 GroupQueryAttention kernel, so the other graphs run the sliding-window layers on the CPU.

In the browser, use import * as ort from 'onnxruntime-web/webgpu': the default onnxruntime-web import cannot run these graphs. On WebGPU, pair decoder_q4f16 with the float32 side graphs of the q4 bundle (vision_encoder_q8, audio_encoder), and keep each packed row under 8,192 tokens, or at most 4,096 with decoder_q4.

d1-Omni-2B reads its answers from a single forward pass, so there are no sampling parameters to set. More options in the ONNX guide.