d3-edge-GGUF

d3-edge is a 0.59B-parameter multimodal decision model from the vLLM Semantic Router team, including a 0.10B vision encoder, released under Apache-2.0. The card lists Qwen3.5-0.8B as its base model. It takes text or JSON input, optionally with several images and videos, and answers multiple Choice, Yes/No and Score questions about it in a single call. It returns a probability for every option and generates no text. On the official Jev Decision Index 0.3.2 text board it scores 20.2, which is 6.6 points above Decision 2.0's 0.6B model and second among entries of similar size, behind jiwo 0.8B v2 at 25.6. It is not on the text Pareto frontier, because Crivo-150M scores 22.2 with only 0.15B parameters. It scores 31.7 on the vision board, where it is the smallest entry, and 46.6 on the Perception Test video benchmark. It is fast: a text request takes a median of 6.6 ms on one AMD Instinct MI325X, and vLLM Semantic Router's Engine mode with batching serves up to 861 requests/s. You can use it through transformers (AutoModel or a pipeline) or serve it with vllm-sr serve -e vllm-sr/d3-edge.

[!NOTE] This is a Decision Model GGUF that supports the llama.cpp v1/systemone API endpoint.

Model Files

Run with llama.cpp (SystemOne API)

1. Start the Server

Launch llama-server with the GGUF model and wait for the health check endpoint to report ready.

import subprocess, time, requests

MODEL_PATH = "/data/llama.cpp/gguf-new/d3-edge.BF16.gguf"
BASE_URL = "http://127.0.0.1:8080"

server = subprocess.Popen(
    [
        "/data/llama.cpp/build/bin/llama-server",
        "-m", MODEL_PATH,
        "--host", "127.0.0.1",
        "--port", "8080",
        "-ngl", "99",
    ],
    stdout=open("/data/llama-server.log", "w"),
    stderr=subprocess.STDOUT,
)

for _ in range(120):
    try:
        if requests.get(f"{BASE_URL}/health", timeout=2).status_code == 200:
            print("server ready")
            break
    except requests.exceptions.RequestException:
        pass
    time.sleep(2)
else:
    print("server not ready, check /data/llama-server.log")

2. Query the /v1/systemone Endpoint

Send a POST request with a state context and structured questions (choice, noul, or score).

curl -s http://127.0.0.1:8080/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "payments, charges, refunds, invoices",
          "shipping": "delivery, tracking, lost or late parcels",
          "technical": "bugs, errors, login problems"
        }
      },
      "angry": {
        "type": "noul",
        "instructions": "Is the customer angry?"
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["can wait", "this week", "today", "right now"]
      }
    }
  }' | python3 -m json.tool

3. Example Response

{
    "model": "/data/llama.cpp/gguf-new/d3-edge.BF16.gguf",
    "answers": {
        "route": {
            "type": "choice",
            "choice": "billing",
            "probabilities": {
                "billing": 0.9663653658895354,
                "shipping": 0.00875909422452679,
                "technical": 0.02487553988593776
            },
            "confidence": 0.8484932369775011
        },
        "angry": {
            "type": "noul",
            "noul": 0.6184953642530783
        },
        "urgency": {
            "type": "score",
            "score": 0.35465875921727147,
            "legend": {
                "0": "can wait",
                "1": "this week",
                "2": "today",
                "3": "right now"
            },
            "probabilities": {
                "0": 0.8317821963345012,
                "1": 0.06539879827289738,
                "2": 0.019197055233429922,
                "3": 0.08362195015917141
            },
            "confidence": 0.5564073180589668
        }
    },
    "usage": {
        "input_tokens": 350,
        "output_tokens": 0
    }
}

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods.