d3-turbo-INT4

d3-turbo-INT4 is d3-turbo, the upgraded 27B multimodal decision model of Decision 3.0, with its decoder's projections in int4. Give it an input (text or JSON, optionally with images and videos) and the questions you need answered: pick one of several options, say yes or no, rate on a scale, or point to where something is in an image or a video. It reads the input once per request, answers every question on it and returns a probability for every answer, as d3-turbo does, in 16.6 GB of weights instead of 52.2 GB.

Highlights

• 71.39 on the Jev Decision Index 0.3 public suite against d3-turbo's 71.62, every area within 0.5 points, with d3-turbo's top answer on 98.4% of 272,229 questions.
• 75.81 on the public vision suite against 76.64 for d3-turbo on the same GPU, with d3-turbo's top answer on 94.4% of 12,210 questions.
• 0.91× to 1.00× d3-turbo's speed per request on one AMD Instinct MI300X, depending on the kind of request (text 148 ms against 135 ms at the median), and 0.97× its text throughput under load. The int4 layers unpack their weights to BF16 for every product, so they save memory rather than time. For speed, use d3-turbo-FP8.
• 16.6 GB of weights instead of 52.2 GB; 46.4 GiB of GPU memory at most in the speed benchmark against 78.3 GiB.
• Box, images in options and tracking within 2 points: Box accuracy at IoU 0.5 is 0.885 against 0.896, images in questions and options 0.949 against 0.957, tracking success AUC 0.778 / 0.782 against 0.796 / 0.807.

Quickstart

pip install "transformers==5.17.0" "compressed-tensors>=0.17" torch torchvision triton pillow opencv-python-headless safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers

import sys

from huggingface_hub import hf_hub_download, snapshot_download

path = snapshot_download("vllm-sr/d3-turbo-INT4")
sys.path.insert(0, path)
from d3q.load import load_d3

model = load_d3(path, device="cuda:0")


def asset(name):
    return hf_hub_download("vllm-sr/d3-turbo", f"assets/{name}")


# Text: several questions, one call
print(model.system_one(
    state="Ticket: the invoice total is wrong after the plan change.",
    questions={
        "route": {"type": "choice", "instructions": "Which team handles this?",
                  "criteria": {"billing": "Billing", "tech": "Technical support"}},
        "urgent": {"type": "noul", "instructions": "Is the customer asking for an urgent fix?"},
    },
))

# Box: where is it in the image? Four questions, one read of the photo
print(model.system_one(
    state="A photo from a dog festival.",
    images=[asset("example-frisbee.jpg")],  # local paths, http(s) URLs, PIL images or base64-encoded images
    questions={
        "frisbee": {"type": "box", "instructions": "The frisbee"},
        "tail": {"type": "box", "instructions": "The dog's tail"},
        "cone": {"type": "box", "instructions": "The orange cone"},
        "camera": {"type": "box", "instructions": "The camera on the tripod"},
    },
))

# Images in options: each option is an image that only this question sees
print(model.system_one(
    state="This animal: <animal>",
    questions={"pick": {"type": "choice", "instructions": "Which picture shows the same kind of animal?",
                        "criteria": {"a": "<a>", "b": "<b>", "c": "<c>"}}},
    assets={"<animal>": asset("example-panda.jpg"), "<a>": asset("example-option-a.jpg"),
            "<b>": asset("example-option-b.jpg"), "<c>": asset("example-option-c.jpg")},
))

# Video: a track over the whole clip, two boxes per second
print(model.system_one(
    state="A bird at a feeder.",
    videos=[asset("example-hummingbird.mp4")],  # local paths, http(s) URLs, base64-encoded videos or frame arrays
    questions={"target": {"type": "track", "instructions": "The hummingbird", "rate": 2}},
))

load_d3 is d3-turbo's own runtime (d3_runtime.py) with its backbone loader swapped for d3q/: transformers loads the checkpoint with its weights packed, and each quantized projection becomes a layer that unpacks its int4 weight to BF16 with one Triton kernel and multiplies in BF16. Prompting, reading the input once, the FP32 readout, Box and Track answers, images and videos are d3-turbo's own; the answers are formatted as on d3-turbo's card. AutoModel.from_pretrained(..., trust_remote_code=True) alone does not run int4 weights.

Checked on one AMD Instinct MI300X with transformers 5.17.0 and compressed-tensors 0.17.0: the four requests above are answered as d3-turbo answers them through the same path (billing; each edge of the frisbee's, the tail's and the cone's boxes within 0.007 of d3-turbo's; picture b at 0.999 against 1.000; a 22-point track for both), except the camera on the tripod, which d3-turbo already finds with low confidence (0.25): this package gives it 0.05, with its bottom edge at 0.837 against 0.756.

Run with vLLM Semantic Router

The vLLM Semantic Router model runtime loads d3-turbo-INT4 as it loads d3-turbo, verifies the package against its MODEL_MANIFEST.json, and keeps the weights packed in int4 on the GPU (#5066):

pip install -e "src/model-runtime[multimodal]" flash-linear-attention   # in a checkout of vllm-project/semantic-router
vllm-srun serve vllm-sr/d3-turbo-INT4 --revision 6a5282347853a4bfb39e6762a653bef6b5f06140 --port 8899 --max-request-bytes 67108864

curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
  "model": "vllm-sr/d3-turbo-INT4",
  "state": "Write a Python function that reverses a string.",
  "questions": {
    "kind": {"type": "choice", "instructions": "What kind of work is this?",
             "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}},
    "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"}
  }
}'

vllm-srun takes a Hub package that is not built in only at a full 40-hex --revision; requests with videos need --max-request-bytes above the default 8 MiB. The Box, images-in-options and tracking results below come from vllm-srun serve run on a local copy of this repository (the same weights and loaded files). Serving it from the Hub at the revision above was checked with the request above and a Box request; engine mode (vllm-sr serve vllm-sr/d3-turbo-INT4 --engine) has not been tried with this repository yet.

Serving performance

By input length, by number of questions, by area and family

By area the text speed-up is 0.84× to 0.97× (arts 0.85×, knowledge 0.91×, language 0.93×, retrieval 0.84×, tools 0.97×); by image benchmark 0.91× (Moderation, R-Bench-M) to 1.01× (Winoground); by Box family 0.88× to 0.95×; video tracking 1.00× at 1 update per second and 0.99× at 2.

Earlier, separate timings on the same GPU: one question per request on the tasksource test split below, one request at a time, median 57.4 ms against 42.3 ms, p90 110.7 against 96.3 ms, p95 145.2 against 133.8 ms (2026-10-24, d3-turbo on 2026-10-20).

• Throughput under load does not grow past 4 requests at a time for any of the three packages with these requests, and latency grows with the queue.
• Why INT4 is not faster here: the runtime's fused int4 kernel runs on CUDA only, so on ROCm every int4 projection unpacks its weights to BF16 before a BF16 product. That cost is fixed per forward, so short requests feel it most (0.73× under 128 tokens) and requests of 4,000 tokens or more not at all (1.00×).
• GPU memory is the device's used memory (rocm-smi), PyTorch's cached blocks included, with requests of up to about 15,000 tokens.
• The benchmark is the fixed set of 5,000 requests in the dataset vllm-sr/d3-perf-bench (private to the vLLM Semantic Router team; text across seven input lengths and the Decision Index's five areas, images, Box, ask once and video), which times any d3-turbo package on the same requests.

_{Measured on 2026-10-25 on one AMD Instinct MI300X (amdgpu 6.19.14), ROCm 7.2 image with PyTorch 2.12, Triton 3.7.0 and FLA 0.5.2, with the vLLM Semantic Router model runtime from main as of 2026-10-20 and its default settings (--max-request-bytes 67108864). Each package was served alone on the GPU, one after the other: every request one at a time, then the text requests 4, 16 and 64 at a time.}

Evaluation

Every comparison is against d3-turbo (BF16) on the same questions. The text index compares with d3-turbo v1.1's own run of the public suite; images, Box, images in options and tracking compare with d3-turbo run on the same MI300X through the same engine. Through the same server, requests and scoring, d3-turbo gives the same hit or miss as its own evaluation rows on 99.5% of the Box items and the same right or wrong answer on 99.7% of the images-in-options items, so the differences below are the quantization's.

Text: Jev Decision Index 0.3, public suite

All 140,178 requests of the public suite, scored by the official 0.3 kit; the index reads the 110,087 requests (272,229 questions) of its 37 benchmarks, and the other 30,091 rows (MMLU, RouterBench, SGD, ARC, SimpleBench) are not part of it. The d3-turbo column is d3-turbo v1.1's own run on an AMD Instinct MI325X, which d3-turbo's card reports as 71.7; scored by the kit this card's run used, its results give 71.62. This package gives d3-turbo's top answer on 98.37% of the index's questions (4,450 change); the lowest agreement is on GPQA Diamond (84.2%), ChessBench legal moves (84.3%) and HLE text (84.8%). KL(d3-turbo ‖ this package): mean 0.0038, median 0.00011, 95th percentile 0.015, 99th percentile 0.068.

Images: Jev Decision Index 0.3, public vision suite

By benchmark (accuracy)

All 12,210 rows, both packages through the same engine on the same MI300X. d3-turbo's card gives its vision board estimate (72.9), which is not this public index. This package gives d3-turbo's top answer on 94.40% of the questions (684 change), most of the changes on MMMU-Pro vision (86.4% agreement) and R-Bench-M (87.7%); every other benchmark agrees on 92.4% to 99.4%. KL: mean 0.020, median 0.0025, 99th percentile 0.26.

Images: Box and images in options

d3-turbo's own Box test set (5,836 items) and images-in-options test set (2,752 questions), scored as d3-turbo's evaluation scores them. On Box this package hits or misses as d3-turbo does on 98.0% of the items (91 hits become misses, 27 misses become hits): −1.1 points at IoU 0.5 (95% interval −1.5 to −0.7), and at IoU 0.75 0.7546 against 0.7939 (mean IoU 0.7929 against 0.8152). The photos lose the most (COCO spatial 0.705 → 0.684, unique 0.828 → 0.814, exemplar 0.731 → 0.710) and receipts 0.996 → 0.976; screens, charts, shapes and tables stay within 0.008. On images in options it is right or wrong as d3-turbo is on 98.4% of the questions (32 right answers become wrong, 12 wrong ones right): −0.7 points (95% interval −1.2 to −0.3), most of it in the compose-frames family (0.943 → 0.908).

Videos: tracking

The 96 tracking test clips of d3-turbo of at most 60 s (2,690 moments at rate 1, 5,377 at rate 2), sent as their own JPEG frames muxed into Matroska at the clip's frame rate without re-encoding. At rate 1, the tracks of the three targets of one clip (120 moments) come back with times that do not line up with the test's moments for both packages and count as misses. This package hits or misses as d3-turbo does on 97.6% and 96.6% of the moments with a visible target, and calls the target visible or not as d3-turbo does on 97.5% and 97.2%.

Text: tasksource-jev-typed-decisions, test split

Every usable row of the test split of tasksource/tasksource-jev-typed-decisions at revision 28c39125 (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, 368 answers changed (97.38%), answered by the vLLM Semantic Router model runtime on one AMD Instinct MI300X. Agreement compares the top answer of every question with d3-turbo on the same requests (Choice and Score: the highest-probability option; Yes / No: p ≥ 0.5). An answer is correct when it is the request's target (Yes / No: the target's side of 0.5). Accuracy is given per type only, since the three types count a correct answer differently. The text calibration prompts come from the train split of the same dataset, and none of their states appears in the test split, so this checks the package on the distribution it was calibrated on; the Jev Decision Index suites above are the independent check.

KL by type and Typed Decisions

KL(d3-turbo ‖ this package) on the tasksource test split, mean / median: Choice 0.00675 / 0.000301, Yes / No 0.00353 / 0.000104, Score 0.00651 / 0.000810.

Typed Decisions (LocalLLaMA/typed-decisions, all test split at d9eda7f3: 400 cases of five questions, one /v1/systemone request per case, scored as the dataset's bench/score_with_jobs.py scores them; the runtime gives a Yes / No answer as P(true), which is turned into the true / false pair the scorer reads). Both columns ran through the model runtime on the MI300X above, so they differ only in the weights; the dataset's leaderboard lists d3-turbo at 0.736 accuracy (KL 0.438) from its own submission.

How the package was chosen

An earlier build of this package, calibrated on the 512 text prompts alone in groups of 128, kept the text and vision indexes close to d3-turbo but hit only 0.436 at IoU 0.5 on the Box test set (tracking success AUC 0.539 / 0.534). Rounding to int4 to nearest loses Box entirely, rounding to int8 keeps it, and the loss builds up over the decoder rather than in one place, so this build adds Box and images-in-options requests to the calibration data and tries two group sizes.

The package was chosen on development data, not on the test splits above, by three rules set before any candidate was measured: lose at most 4 points of Box accuracy on the Box development split, at most 1 point of accuracy on the images-in-options development split, and change no more of d3-turbo's answers on the train rows below than the text-only build. The train rows are 10,000 rows of the first train shard that share no row with the 128-prompt set used earlier, minus the 233 whose state is one of the 512 text calibration prompts (9,767 rows). The Box development split was sampled at 60 items per family (480), the images-in-options one at 30 questions per family (180), run through d3q.

Groups of 128 failed the third rule; groups of 64 passed all three and were then run once on the test splits above.

_{Built on 2026-10-24 by awq.py and calib_mixed.py in bench/d3-quant-eval on one AMD Instinct MI300X. For each group of projections that read one input (q/k/v_proj; in_proj_qkv/z with in_proj_a/b; gate/up_proj; down_proj), a per-input-channel scale x^a / w^(1-a) is searched over a in [0, 1) to minimise the squared error of the decoder layer's output with those projections rounded to int4, measured on 96 of the shuffled calibration requests (40 of them with images), and folded into the norm or up_proj before it; o_proj and linear_attn.out_proj take no scale. A large scale folded into d3-turbo's RMSNorm (which multiplies by 1 + w) is capped so that 1 + w keeps two BF16 steps, and the projections after the norm take the scale the stored w gives, so the fold leaves the BF16 model unchanged. GPTQ then rounds each projection column by column, spreading each column's rounding error over the columns not yet rounded through the inverse Hessian of its smoothed inputs over all 832 calibration requests (1% dampening), with each group's scale and zero point taken from its already compensated weights. The Hessian is accumulated as the requests pass, rather than from stored inputs, and gives the same rounding. The text calibration prompts are 512 rows of the first train shard (state of at most 6,000 characters, revision 28c39125); the Box and images-in-options requests are rows of the train splits of d3-turbo's own Box and images-in-options training data, from which every row that shares an image with a dev or test split was dropped. Evaluation runs, all on one AMD Instinct MI300X: the Decision Index text suite with the official 0.3 kit (decision-index f5a6e4f, as d3-turbo v1.1's own run) through d3q, one request at a time; the public vision suite with the same kit; Box, images in options, tracking, the speed benchmark and Typed Decisions through vllm-srun serve (model runtime from main as of 2026-10-20) at its default settings.}

License

Apache-2.0 (LICENSE), as d3-turbo. Built on vllm-sr/d3-turbo.

Citation

@misc{d3turbo_2026,
  title        = {{d3-turbo}: A Multimodal Decision Model that Reads Once},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-turbo}}
}