d3-W4A16-GPTQ

d3-W4A16-GPTQ is d3, the 27B multimodal foundation decision model of Decision 3.0, with its decoder's projections in int4. Give it an input (text or JSON, optionally with images and videos) and the questions you need answered: pick one of several options, say yes or no, or rate on a scale. It answers them all in one call and returns a probability for every answer, as d3 does, in 16.1 GB of weights instead of 52.2 GB.

Highlights

• Scores 65.15 on the Jev Decision Index 0.3 public suite against d3's 65.45, giving d3's top answer on 97.93% of its 312,320 questions; every area within 0.5 points of d3.
• Keeps 97.98% of d3's answers on the 14,031 test requests of tasksource-jev-typed-decisions, with accuracy 83.10% against d3's 83.25%.
• 16.4 GB of GPU memory for the weights instead of 52.2 GB with the vLLM Semantic Router model runtime, on one AMD Instinct MI300X and on one NVIDIA RTX PRO 6000 alike.
• 1.28× to 1.47× d3's latency on the same GPU: 61.5 ms against 47.9 ms per request on one AMD Instinct MI300X and 91.2 ms against 61.9 ms on one RTX PRO 6000 (median, graphs replaying, model runtime). The int4 layers unpack their weights to BF16 for every product, so they save memory rather than time.
• Reads images and videos as d3 does: the vision tower is unchanged. On the vision board's public suite it scores 73.55 against d3's 74.15 on the same machine and gives d3's top answer on 95.69% of its 12,210 questions.
• Perception Test: 76.94% on all 19,140 validation questions against d3's 77.73% on the same machine, giving d3's top answer on 95.13% of them.

Quickstart

pip install "transformers==5.17.0" "compressed-tensors>=0.19" torch torchvision triton pillow opencv-python-headless safetensors accelerate
pip install flash-linear-attention causal-conv1d  # optional: fast GPU kernels for the linear-attention layers

import json
import sys

from huggingface_hub import hf_hub_download, snapshot_download

path = snapshot_download("vllm-sr/d3-W4A16-GPTQ")
sys.path.insert(0, path)
from d3q.load import load_d3

model = load_d3(path, device="cuda:0")

# Text
result = model.system_one(
    state="The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "returns": "Refunds, replacements and damaged deliveries",
                "billing": "Payments, invoices and charges",
                "technical": "Product setup and faults"
            }
        },
        "receipt": {
            "type": "noul",
            "instructions": "Does the customer have a receipt?"
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this request?",
            "criteria": [
                "Routine",
                "Soon",
                "Today"
            ]
        }
    },
)
print(json.dumps(result["answers"], indent=2))

# Text and an image (or several)
receipt = hf_hub_download("vllm-sr/d3", "assets/example-receipt.png")
result = model.system_one(
    state="The customer says the blender arrived cracked and attached the receipt.",
    images=[receipt],  # local paths, http(s) URLs, PIL images or base64 data URLs
    questions={
        "on_receipt": {
            "type": "noul",
            "instructions": "Does the receipt list the blender?"
        }
    },
)
print(json.dumps(result["answers"], indent=2))

# Text and a video (or several)
clip = hf_hub_download("vllm-sr/d3", "assets/example-video.mp4")
result = model.system_one(
    state="A short clip from a test camera.",
    videos=[clip],  # local paths, http(s) URLs, base64 data URLs or frame arrays
    questions={
        "color_change": {
            "type": "noul",
            "instructions": "Does the square change color?"
        }
    },
)
print(json.dumps(result["answers"], indent=2))

load_d3 is d3's own runtime (d3_runtime.py) with its backbone loader swapped for d3q/: transformers loads the checkpoint with its int4 weights still packed (dequantize=False), and each quantized projection becomes a layer that unpacks its weights to BF16 with one Triton kernel and multiplies in cuBLAS, bit for bit as compressed-tensors would dequantize them. Prompting, noncausal full attention, the FP32 readout, images and videos are d3's own; images and videos are read as d3 reads them. AutoModel.from_pretrained(..., trust_remote_code=True) alone dequantizes the whole checkpoint to BF16 and needs about 52 GB of GPU memory again.

Run with vLLM Semantic Router

vLLM Semantic Router serves d3-W4A16-GPTQ in Engine mode, which runs one model behind the System One API on your own machine, as it serves d3. It verifies the package against its MODEL_MANIFEST.json, as it verifies d3, and keeps the weights in int4 on the GPU, with its own int4 layers rather than d3q/.

curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev --no-launch
export PATH="$HOME/.local/bin:$PATH"
vllm-sr serve -e vllm-sr/d3-W4A16-GPTQ

The platform is detected automatically; choose one with --platform auto|cpu|cuda|rocm (default auto), and a commit with --revision. Int4 needs no special matrix units, only about 17 GB of free GPU memory (see Needs above). Then ask questions about a request:

The first start downloads the package (about 16 GB) before loading it; if it stops with did not become ready within 10m0s, run the same command again: the download is kept and the second start only loads.

curl -s localhost:8899/v1/systemone -H 'content-type: application/json' -d '{
  "model": "vllm-sr/d3-W4A16-GPTQ",
  "state": "Write a Python function that merges two sorted lists.",
  "questions": {
    "kind": {"type": "choice", "instructions": "What kind of work is this?",
             "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}},
    "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"}
  }
}'

Images and videos go in images and videos as for d3. vllm-sr stop stops it. See Router and Engine modes for limits and options.

Serving performance

On one NVIDIA RTX PRO 6000, the model runtime answers a request in 91.2 ms against d3's 61.9 ms (median, one request at a time, graphs replaying, both in one process), in 16.4 GB of GPU memory against 52.2 GB.

• Graphs: replaying the forward as a graph (#4999) changes little on this GPU, for d3 and for this package alike: without graphs the same requests take 92.5 ms against d3's 62.5 ms. On the MI300X graphs help both: d3 from 69.2 to 47.9 ms and this package from 83.6 to 61.5 ms, so the gap to d3 is 1.28× with graphs and 1.21× without.
• Kernels: one int4 projection with its unpacking takes 1.1× to 1.4× the time of the BF16 GEMM at 64 to 1,024 rows on the MI300X: the int4 layers save memory, not time. Under the runtime's approximate profile (max_speed), NVIDIA GPUs multiply straight from the packed weights for inputs of up to 256 rows instead.

_{Latency and memory measured on 2026-10-16, one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (driver 595.99.02): PyTorch 2.11.0+cu128, Triton 3.6.0, FLA 0.5.2, causal-conv1d 1.7.0; vLLM Semantic Router model runtime of #5066 (da21a679) with the graph capture of #4999 (02423e85), exact profile. Requests: the 1,000 train rows of tasksource-jev-typed-decisions in the #5066 record, one question each. Both packages loaded in one process; every request answered by both in turn, alternating which goes first, the device synchronized around each: the first 500 rows answered a third time so that their shape's graph replays, or all 1,000 a second time without graphs. Memory: allocated after loading and at the peak of the 20 longest of those rows (up to 5,928 characters of state), each package in its own process. Kernels measured on 2026-10-14, one AMD Instinct MI300X: ROCm image with PyTorch 2.12, Triton 3.7.1 and FLA 0.5.2, one of each of d3's projection shapes with random weights, triton.testing.do_bench. MI300X rows: measured on one AMD Instinct MI300X with the same vLLM Semantic Router model runtime, requests, passes and memory measurement.}

Evaluation

Text: Jev Decision Index 0.3, public suite

The five areas are the index's areas (chance-corrected skill; the public index averages them with fixed weights), with differences from d3 in parentheses. Agreement compares the top answer of every question with d3's official run (choice and score: the highest-probability option; yes / no: p ≥ 0.5): 312,320 questions in all, and per area the questions of that area's benchmarks.

The complete public suite (every request of d3's official run), run with the official kit (apolinario/decision-index f8552a1) through d3q/ and scored by it, against d3's official run in vllm-sr/d3-decision-index; rescoring that run with the same kit reproduces 65.45 exactly. The board's full score (66.3 for d3) also needs private tests that only the maintainers run, so it is not measured here: compare with d3's public 65.45, not 66.3.

Also on the test split of tasksource/tasksource-jev-typed-decisions:

Every usable row of the split at revision 28c39125 (a state of at most 6,000 characters, no repeated options): 14,031 requests of one question each, answered by the vLLM Semantic Router model runtime on one AMD Instinct MI300X. Agreement compares the top answer of every question with d3 on the same requests (choice and score: the highest-probability option; yes / no: p ≥ 0.5). Two calibration states, an LSAT logic-game passage and a news article, are also the states of 2 of these 14,031 requests.

The evaluation results listed on this page (LocalLLaMA/typed-decisions) are a separate run: that benchmark's own test split scored with its score_model.py, accuracy 0.751 against d3's 0.7485 on the same setup (the vLLM Semantic Router model runtime on one AMD Instinct MI300X).

Images: Jev Decision Index vision board 0.3.2

By benchmark (chance-corrected skill, %)

All 12,210 rows of the kit's public vision suite (11 benchmarks), one image request per row, run with the official kit (apolinario/decision-index f8552a1) through d3q/ and scored by it. d3 (BF16) answered the same rows the same way on the same machine and scores 74.15, in line with the official vision board's 74.1. Agreement compares the top answer of every question with that BF16 run.

Videos: Perception Test

By area (accuracy, %)

Every multiple-choice question of Perception Test validation (19,140 questions over 5,260 videos, three options each, chance 33.3): one request per video with all its questions and the state "A video.", videos read with d3's defaults, through d3q/ on one RTX PRO 6000. d3 (BF16) answered the same requests the same way on the same machine and scores 77.73; d3's card reports 77.4 from its internal evaluation, whose prompt may differ.

_{Record: decision3-quantized.md in #5066. Quantized with llm-compressor 0.14.0 and compressed-tensors 0.19.0 (torch 2.11.0+cu128, transformers 5.17.0) on one GeForce RTX 5090, calibrating the text model layer by layer with the full-attention layers under the noncausal key mask of d3_runtime.PromptEncoder, as at inference. The calibration prompts are text only and none of their states appears in the Decision Index public suite. QUANT_INFO.json records the recipe, versions and calibration data; MODEL_MANIFEST.json is d3's manifest with the hashes, sizes and model identity of this package's own files, written by write_manifest of bench/d3-quant-eval/awq.py. d3q_engine.py is the Decision Index engine for the kit (--engine d3q_engine:D3QEngine --option model=<this folder>).}

License

Apache-2.0 (LICENSE), as d3. Built on vllm-sr/d3, which is built on Qwen/Qwen3.8-27B (Apache-2.0).

Citation

@misc{d3_2026,
  title        = {{d3}: A Multimodal Foundation Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3}}
}