Decision-2.0-Lux-9B · GGUF for llama.cpp hidden-state serving

This repository is a llama.cpp build of vllm-sr/Decision-2.0-Lux-9B, the 9B decision model of Decision 2.0 (Qwen3.5 backbone with a candidate head). It holds everything needed to serve the Jev/SystemOne-compatible /v1/systemone API (choice, noul, score questions, a probability for every option) on a single 8 GB GPU, including GPUs without bf16 such as Turing.

The original package runs in Transformers with Triton kernels under bf16 autocast and stores its backbone in fp32 (32 GB). Here the backbone is a 4-bit GGUF; the head, the prompt encoding and the answer format are the original package code, unchanged.

Not for text generation. The model has no trained LM head. output.weight in the GGUF is a Q2_K copy of the token embeddings, present only so llama.cpp does not place a second copy of the embeddings on the GPU. It is never used.

Files

How it runs

Decision 2.0 asks every question as its own prompt: Context:\n<state>\n\nTask type: …\nQuestion:\n…\nOptions: + one <option> block per option + Decision:. The head reads the backbone's final hidden state (after the final norm) at the last token of every option and at the last token of the prompt, and scores the options from those vectors.

request → decision2 encode()   (original code: one prompt per question, option end positions)
        → dec2_bridge           llama.cpp; the questions' common token prefix (the state) is decoded once,
                                its sequence state is saved and restored for every question, which then
                                decodes only its own suffix → hidden states at the needed positions only
        → CandidateHead (fp32)  original head → product_answer() (original answer format)

Decoding the shared prefix once is exact, not an approximation: the questions start with the same tokens, so the backbone computes the same states for them either way. The original package's share_context option is a different, approximate mechanism; with this server a 5-question request over a 2.5k-token state costs little more than a 1-question request.

Build and run. Tested with llama.cpp 8212c78 (Qwen3.5 support, nextn output, load_mode):

g++ -O2 -std=c++17 -I llama.cpp/include -I llama.cpp/src -I llama.cpp/ggml/include serving/dec2_bridge.cpp \
    -L llama.cpp/build/bin -lllama -lggml -lggml-base -o dec2_bridge
pip install torch numpy safetensors "transformers>=5"
MODEL_DIR=. GGUF=./Decision-2.0-Lux-9B-Q4_K_M-dyn.gguf BRIDGE=./dec2_bridge PORT=8012 python3 serving/lux_api.py

curl -s localhost:8012/v1/systemone -d '{"model": "lux",
  "state": "The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this request?",
                "criteria": {"returns": "Refunds, replacements and damaged deliveries",
                             "billing": "Payments, invoices and charges", "technical": "Product setup and faults"}},
    "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"},
    "urgency": {"type": "score", "instructions": "How urgent is this request?", "criteria": ["Routine", "Soon", "Today"]}}}'

Hidden states come from llama.cpp's nextn output (llama_set_embeddings_nextn), not from embeddings = true, which at this commit also reserves n_vocab logits per token in pinned host RAM. Weights are read into device buffers (load_mode = none), so the 6 GB file is not kept mapped. The head runs in fp32 (it is 4.5M parameters).

Quantization

• Source: the original fp32 backbone, converted to a bf16 GGUF with llama.cpp's convert_hf_to_gguf.py (serving/convert_lux_gguf.py: --no-mtp, tensor names mapped to model.*, separate output.weight). The original runtime itself holds the backbone's Linear weights in bf16, so the bf16 GGUF loses nothing that the original GPU path keeps.
• Base type: Q4_K_M, with these per-tensor overrides (serving/quant_layout.tt, the layout of current llama.cpp for Qwen3.5):
  • ssm_alpha, ssm_beta (gated delta net gates): F32, unquantized.
  • ssm_out and the full-attention attn_k/attn_v/attn_output: Q8_0 (ssm_out of the middle layers Q6_K).
  • Extra bits in the first and last layers (ffn_up 0–7, attn_gate, attn_qkv, ffn_down 0/27/31).
  • token_embd: Q8_0. llama.cpp keeps it in host RAM, so it costs no VRAM.
  • output: Q2_K. Unused, see above.
• Importance matrix: 1.5M tokens at context 2048, every one rendered in Decision 2.0's own prompt format (serving/render_calibration.py): 2,701 prompts from 5,018 decision records with 1–3 questions of all three types in English and Czech. States come from the train splits of BoolQ, AG News and SST-2, multi-article AG News digests, Czech news (CIIRC-NLP/czech_news_simple-cs, hynky/czech_news_dataset_v2) and bartowski's calibration_datav3. None of the evaluation items below are in it.

Measured quality

The port. The bf16 GGUF through this server against the original package (Transformers, exact path) on 186 questions of all three types, including Czech and states up to 4k tokens: 100 % the same decisions, mean |Δp| 0.0007, max |Δp| 0.015, KL 0.00001.

The quantization. Each GGUF ran through the full pipeline above and was compared with the same pipeline on the bf16 GGUF. Items: 900 questions, 300 each from BoolQ (validation), AG News (test) and SST-2 (validation), plus 24 long synthetic states (2k–11k tokens) with 5 questions each. KL is the mean KL(bf16 ‖ quant) per question; agreement is the share of questions where both pick the same option. Measured on an NVIDIA GB10:

¹ Every tensor except the token embeddings (host RAM), with a Q2_K output.weight.

The layout and the decision-format imatrix make this Q4_K_M 6× closer to bf16 by KL than a plain Q4_K_M for 0.5 GB more VRAM. On our own decision tasks (entity-link validation, news routing, proofreading-finding filtering, translation checks; English and Czech) the Q4 build scored within noise of the original package: the same or ±0.01 AUC.

Running on an 8 GB GPU

GPU memory at a 16k context, one sequence, n_ubatch 512 (llama.cpp's buffer sizes): weights 5.0 GB, attention KV 0.5 GB (only 8 of 32 layers use attention), gated-delta-net state 0.05 GB, compute buffer 0.5 GB, plus the CUDA context and the head: about 6.6 GB. Host RAM: the 1.08 GB of token embeddings plus the Python process.

Request latency on an NVIDIA GB10 (median, the whole HTTP request):

How it compares with the other decision models we serve

We run three decision models behind the same /v1/systemone API (Rune and Clef-Flash in production, Lux-9B in testing) and publish all three: Livesport/rune-26b-a4b-v3-GGUF, Livesport/clef-flash-GGUF and Livesport/Decision-2.0-Lux-9B-GGUF. This section is the same in all three cards.

How we measured. 7,580 questions from the validation and test splits of 12 public datasets, English and Czech, sampled with fixed seeds. Every item is one /v1/systemone request with one question; the state and the question text are the same for all three models. Each model ran from the exact file published in its repository, served as its card describes (Rune: llama-server with our adapter, request-scoped prefix reuse; Clef and Lux: the hidden-state bridge), one request at a time on an NVIDIA GB10. Accuracy is the share of questions where the most probable option is the gold label (for the star rating: the exact number of stars).

Reading the table.

• Overall the three are within half a point of each other; they differ by task.
• Rune is best on adversarial reasoning (ANLI, by 12–19 points), on Czech fact checking (CsFEVER) and Czech Facebook sentiment, and it is the best calibrated: its probabilities match observed accuracy most closely.
• Clef-Flash is best on English topic, yes/no-over-a-passage and intent tasks, and on CTKFacts.
• Lux-9B is best on MNLI, the Czech star rating and Czech product reviews, and is close to Clef on the English classification tasks.
• We do not know the training data of the three models in full. Decision models are commonly trained on public NLI, QA and intent data, so results on MNLI, BoolQ or CLINC may be partly in-distribution for some of them.
• On our own (unpublished) English and Czech decision tasks the picture was the same: Rune leads on Czech faithfulness and fact checks, Clef and Lux on English routing and matching.

Speed. Median latency of one request on an NVIDIA GB10, one request at a time:

Every server here runs one request at a time, so more parallel clients add queueing, not throughput.

On a Quadro RTX 4000 8 GB (Turing), measured in production:

• Rune IQ3_S-dyn on 2 GPUs: 0.49 s for ~730 tokens, 1.05 s for ~2.3k tokens, 4.12 s for ~8.9k tokens (1 question; 5 questions add 0.3–0.6 s).
• Clef-Flash Q5_K_M-dyn on 1 GPU: 0.57 s for ~600 tokens, 1.75 s for ~2k tokens, 6.58 s for ~7.4k tokens.
• Lux-9B Q4_K_M-dyn: not deployed on Turing yet. It has the same Qwen3.5-9B-class backbone and serving stack as Clef, so expect similar speed.
• The 9B dense backbones are about 1.6× slower than Rune's mixture of experts on Turing, but need one 8 GB GPU instead of two.

License and credits

• Decision-2.0-Lux-9B by the vLLM Semantic Router team, Apache-2.0, built on Qwen3.5. The head, the decision2 package, the tokenizer and the metadata are their files, unchanged.
• GGUF, quantization and serving code: Livesport, Apache-2.0, provided as is.
• Thanks: bartowski for calibration data.