d3-ultra-NVFP4

d3-ultra-NVFP4 is d3-ultra, the newest 27B multimodal decision model of Decision 3.0, with its decoder's projections in NVFP4 for NVIDIA Blackwell GPUs. Give it an input (text or JSON, optionally with images and videos) and the questions you need answered: pick one of several options, say yes or no, rate on a scale, or point to where something is in an image or a video. It reads the input once per request, answers every question on it and returns a probability for every answer, as d3-ultra does, in 17.2 GB of weights instead of 52.2 GB and in about half the time.

Box loses 1.8 points and the text index 0.8. Box accuracy at IoU 0.5 is 0.854 against d3-ultra's 0.872, and most of the loss is on receipts (0.870 against 0.955). The text index is 70.74 against 71.57, most of it in knowledge (59.7 against 62.3). Images, images in options and tracking stay within 1.5 points. If Box answers on receipts or knowledge questions matter most, use d3-ultra-FP8.

Highlights

• 70.74 on the Jev Decision Index 0.3 public suite against d3-ultra's 71.57: language, retrieval and tools within 0.7 points, arts −1.3, knowledge −2.6, with d3-ultra's top answer on 97.0% of 272,229 questions.
• 74.45 on the public vision suite against 75.36 for d3-ultra on the same GPU, with d3-ultra's top answer on 93.5% of 12,210 questions.
• Images in options and tracking within 1.5 points: images in questions and options 0.949 against 0.952, tracking success AUC 0.761 / 0.768 against 0.775 / 0.783. Box loses more (see the note above).
• 17.2 GB of weights instead of 52.2 GB, and about twice as fast: 32.5 ms against 63.4 ms at the median for one text question at a time on one NVIDIA RTX PRO 6000.

Quickstart

pip install "transformers==5.17.0" "compressed-tensors>=0.19" "torch>=2.11" torchvision triton pillow opencv-python-headless safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers

import sys

from huggingface_hub import hf_hub_download, snapshot_download

path = snapshot_download("vllm-sr/d3-ultra-NVFP4")
sys.path.insert(0, path)
from d3q.load import load_d3

model = load_d3(path, device="cuda:0")


def asset(name):
    return hf_hub_download("vllm-sr/d3-ultra", f"assets/{name}")


# Text: several questions, one call
print(model.system_one(
    state="Ticket: the invoice total is wrong after the plan change.",
    questions={
        "route": {"type": "choice", "instructions": "Which team handles this?",
                  "criteria": {"billing": "Billing", "tech": "Technical support"}},
        "urgent": {"type": "noul", "instructions": "Is the customer asking for an urgent fix?"},
    },
))

# Box: who is in the canoe? Three questions, one read of the painting
print(model.system_one(
    state="A painting of fur traders on a river.",
    images=[asset("fur-traders.jpg")],  # local paths, http(s) URLs, PIL images or base64-encoded images
    questions={
        "cat": {"type": "box", "instructions": "The cat"},
        "canoe": {"type": "box", "instructions": "The canoe"},
        "dog": {"type": "box", "instructions": "A dog"},
    },
))

# Photo to etching: the question holds a photo, each option is an etching
print(model.system_one(
    state="An architecture quiz.",
    questions={"pick": {"type": "choice", "instructions": "Which 18th-century etching shows the same building as <building>?",
                        "criteria": {"a": "<a>", "b": "<b>", "c": "<c>", "d": "<d>"}}},
    assets={"<building>": asset("pantheon.jpg"), "<a>": asset("pantheon-a.jpg"), "<b>": asset("pantheon-b.jpg"),
            "<c>": asset("pantheon-c.jpg"), "<d>": asset("pantheon-d.jpg")},
))

load_d3 is d3-ultra's own runtime (d3_runtime.py) with its backbone loader swapped for d3q/: transformers loads the checkpoint with its FP4 weights still packed, and each quantized projection becomes a layer that quantizes its input to NVFP4 with one Triton kernel and multiplies on the FP4 tensor cores in torch.nn.functional.scaled_mm. Prompting, reading the input once, the FP32 readout, Box and Track answers, images and videos are d3-ultra's own; the answers are formatted as on d3-ultra's card. AutoModel.from_pretrained(..., trust_remote_code=True) alone dequantizes the whole checkpoint to BF16 and needs about 52 GB of GPU memory again.

Checked on one NVIDIA RTX PRO 6000 Blackwell with transformers 5.17.0 and compressed-tensors 0.19.0: the three requests above are answered as d3-ultra answers them through its own runtime, but for one Box question. Every edge of the cat box is within 0.004 of d3-ultra's and of the canoe box within 0.003, etching d gets 0.998 against 0.996, and the ticket goes to billing (urgent: 0.12 against 0.10). The one difference is the dog, which is not in the painting: d3-ultra gives it confidence 0.00, this package 0.67, so a confidence threshold of 0.5 would report a dog here.

Run with vLLM Semantic Router

The vLLM Semantic Router model runtime loads d3-ultra-NVFP4 as it loads d3-ultra, verifies the package against its MODEL_MANIFEST.json, and keeps the weights in NVFP4 on the GPU, multiplying on the FP4 tensor cores:

pip install -e "src/model-runtime[multimodal]" flash-linear-attention   # in a checkout of vllm-project/semantic-router
vllm-srun serve vllm-sr/d3-ultra-NVFP4 --revision cf65de7ce81062b4af720636651c4a2b99353ddb --port 8899 --max-request-bytes 67108864

curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
  "model": "vllm-sr/d3-ultra-NVFP4",
  "state": "Write a Python function that reverses a string.",
  "questions": {
    "kind": {"type": "choice", "instructions": "What kind of work is this?",
             "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}},
    "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"}
  }
}'

This package was served with the model runtime from main at b88ca01b (2026-10-29). A runtime that predates d3-ultra's prompt layout and per-request temperature refuses the package, since the model identity in its manifest differs. vllm-srun takes a Hub package that is not built in only at a full 40-hex --revision; requests with videos need --max-request-bytes above the default 8 MiB. Every result below comes from the model runtime run on a local copy of this repository (the same weights and loaded files).

One question per request on the tasksource test split below, one request at a time through the model runtime on one NVIDIA RTX PRO 6000 Blackwell: median 32.5 ms against 63.4 ms for d3-ultra.

Until 2026-11-02 this repository held an earlier build of this package, calibrated on 175 requests of d3-ultra's own training data (videos included); it stays available at revision becd81ec7ef5b2e16b2eec27519015f312a9238b. How the two compare is under "How the package was chosen" below.

Evaluation

Every comparison is against d3-ultra (BF16) on the same questions. The text index compares with d3-ultra's own public run (vllm-sr/d3-ultra-decision-index); images, Box, images in options, tracking and tasksource compare with d3-ultra run on the same RTX PRO 6000 through the same runtime. d3-ultra's vision run on this GPU gives 75.36 against 75.33 in its public run, so the differences below are the quantization's.

Text: Jev Decision Index 0.3, public suite

The 110,087 requests (272,229 questions) of the 37 benchmarks the index reads, run with the official 0.3 kit through the model runtime and scored by it; re-scored by the same scorer, d3-ultra's public results give the 71.57 of its public run. This package gives d3-ultra's top answer on 97.00% of the index's questions (8,174 change); the lowest agreement is on ChessBench legal moves (70.5%), HLE text (75.5%) and GPQA Diamond (80.1%). KL(d3-ultra ‖ this package): mean 0.020, median 0.00031, 95th percentile 0.079, 99th percentile 0.37. d3-ultra's public run used another engine and GPU, so these differences include theirs as well; on tasksource below, with both packages on the same GPU and runtime, the agreement is 94.5%.

Images: Jev Decision Index 0.3, public vision suite

By benchmark (accuracy)

All 12,210 rows, both packages through the same runtime on the same RTX PRO 6000. This package gives d3-ultra's top answer on 93.51% of the questions (793 change), most of the changes on MMMU-Pro vision (80.6% agreement) and R-Bench-M (81.4%), where d3-ultra itself is right about half the time; every other benchmark agrees on 94.0% (BLINK) to 99.0% (KIE). KL: mean 0.035, median 0.0017, 99th percentile 0.54.

Images: Box and images in options

The Box test set of d3-turbo (5,836 items) and its images-in-options test set (2,752 questions), scored as d3-turbo's evaluation scores them. On Box this package hits or misses as d3-ultra does on 96.4% of the items (158 hits become misses, 54 misses become hits): −1.8 points at IoU 0.5 (95% interval −2.3 to −1.3), and at IoU 0.75 4.7 points lower (mean IoU 0.746 against 0.771). Receipts lose the most (0.955 → 0.870), then COCO spatial (0.653 → 0.641), tables (0.981 → 0.970) and COCO exemplar (0.688 → 0.679); COCO unique 0.801 → 0.798, and charts, screens and shapes stay within 0.006. On images in options it is right or wrong as d3-ultra is on 98.7% of the questions (22 right answers become wrong, 14 wrong ones right): −0.3 points (95% interval −0.7 to +0.1).

Videos: tracking

The 96 tracking test clips of d3-turbo of at most 60 s (2,690 moments at rate 1, 5,377 at rate 2), sent as their own JPEG frames muxed into Matroska at the clip's frame rate without re-encoding. At rate 1, one moment of each of the three targets of one clip comes back 0.25 s off the test's moment for both packages. This package hits or misses as d3-ultra does on 97.2% and 97.3% of the moments with a visible target, and calls the target visible or not as d3-ultra does on 98.5% and 98.0%.

Text: tasksource-jev-typed-decisions, test split

Every usable row of the test split of tasksource/tasksource-jev-typed-decisions (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime one at a time on one NVIDIA RTX PRO 6000 Blackwell. Agreement compares the top answer of every question with d3-ultra on the same requests (Choice and Score: the highest-probability option; Yes / No: p ≥ 0.5). An answer is correct when it is the request's target (Yes / No: the target's side of 0.5). Accuracy is given per type only, since the three types count a correct answer differently. The text calibration prompts come from the train split of the same dataset, and none of their states appears in the test split, so this checks the package on the distribution it was calibrated on; the Jev Decision Index suites above are the independent check. KL(d3-ultra ‖ this package), mean / median: Choice 0.070 / 0.00029, Yes / No 0.025 / 0.00025, Score 0.069 / 0.0025.

How the package was chosen

The first build of this package (here until 2026-11-02, revision becd81ec) was picked among eight NVFP4 builds made with the recipe of d3-turbo-NVFP4 (AWQ, then GPTQ) on 175 calibration requests of d3-ultra's own training data, videos included: different calibration mixes, GPTQ against BF16 or against NVFP4-quantized activations, two GPTQ dampenings and a clipped activation global scale. Six of them were run on d3-ultra's own SR-Box test and the tracking test split above, and the first build was picked among the three that lost least on the mean and the worst change against d3-ultra across SR-Box, d3-ultra's SR-Text test, the public vision suite, and the tasksource, images-in-options and tracking test splits.

Its text agreement on development data stayed below d3-ultra-INT4's rule, so five more builds followed: GPTQ moving the weights only half or a quarter of the way to W' (see the build notes), d3-ultra-INT4's 832 calibration requests (out of memory on the 96 GB GPU), the first 512 of them (this package), and 455 of them with the first build's 57 image and video requests (out of memory). This package was then run on every test of this card and compared with the first build:

On the six tests of the first selection its mean change against d3-ultra is −0.57 points (first build −0.69) and its worst −0.97 (first build −1.22), so it replaced the first build. Both builds were chosen with test splits, not development data alone, unlike d3-ultra-INT4.

The development data of d3-ultra-INT4's selection, measured afterwards for every build: the Box development split (480 items, 60 per family), the images-in-options development split (180 questions, 30 per family) and the top answer on 10,000 rows of the first train shard of tasksource-jev-typed-decisions.

None reaches d3-ultra-INT4's 97.0% agreement rule on the train rows; with activations in FP4 as well, the agreement stays between 94.5% and 96.5%. This package's text calibration prompts come from the same train shard as those 10,000 rows, as d3-ultra-INT4's do.

_{Built on 2026-11-01 by nvfp4_ultra.py, the recipe of vllm-sr/d3-turbo-NVFP4 with GPTQ against the quantized activations, on one NVIDIA RTX PRO 6000 Blackwell. Calibration: d3-ultra-INT4's 832 requests, rebuilt with the selection of its scripts (make_sets.py and calib_mixed.py in bench/d3-quant-eval at 778c3fc9: 512 tasksource text prompts, 32 Box rows per family and 64 images-in-options rows of d3-turbo's training data, every row sharing an image with a development or test split dropped, shuffled), of which the first 512 were used, since the recording of all 832 does not fit in 96 GB of GPU memory: 316 text prompts, 153 Box and 43 images-in-options requests, run through d3-ultra's own runtime as it answers them (its header-v1 layout for text and image tokens included); the AWQ search reads the first 64 (28 with images). AWQ: per decoder layer and group of projections that read one input, the scale s = x^a / w^(1-a) searched over 20 values of a to minimise the layer's output error with that group run as NVFP4 weights times NVFP4 activations, then folded into input_layernorm, post_attention_layernorm and up_proj. GPTQ against the quantized activations: per group, with X its calibration input and Xq its NVFP4-quantized form at the group's input global scale, the weights are first moved to W' = W (Xᵀ Xq)(Xqᵀ Xq)⁻¹, for which W' Xq best matches W X, and W' is then rounded to E2M1 column by column with the Hessian of Xq (1% dampening, blocks of 128 columns), each 16-value block's E4M3 scale taken from its already-compensated weights. q / k / v, in_proj_qkv / in_proj_z and gate / up share their weight and input global scales, so the runtime's merged products apply. Evaluation runs, all on one NVIDIA RTX PRO 6000 Blackwell through the model runtime from main at b88ca01b at its default settings: the Decision Index text and vision suites with the official 0.3 kit; Box, images in options, tracking and tasksource with d3-turbo's evaluation scripts.}

License

Apache-2.0 (LICENSE), as d3-ultra. Built on vllm-sr/d3-ultra.

Citation

@misc{d3ultra_2026,
  title        = {{d3-ultra}: A Multimodal Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-ultra}}
}