Clef-Flash · GGUF for llama.cpp hidden-state serving

This repository is a llama.cpp build of Cloudflare/clef-flash, a 9B decision model (Qwen3.5-9B backbone with a joint schema head). It holds everything needed to serve Clef's Jev/SystemOne-compatible /v1/systemone API on a single 8 GB GPU. Livesport runs it in production that way on a Quadro RTX 4000 (Turing, 8 GB).

Clef does not generate text. Its head reads the backbone's final hidden state of every token and scores every allowed option of every question in one forward pass. The GGUF here is therefore quantized for that job: its bits go into the backbone, not into the LM head.

Not for text generation. In these GGUFs output.weight (the LM head) is deliberately stored at Q2_K, because the Clef pipeline never uses its logits. The head takes the option tokens' rows from lm_head_f16.safetensors instead. The VRAM saved goes into a Q5_K_M backbone, which then needs 5.6 GB on the GPU: 0.3 GB more than a standard Q4_K_M and 0.2 GB less than a standard Q5_K_S, and closer to bf16 than either.

Files

How it runs

request → encode_record()      (joint_schema_model.py: tokenization, question/option spans)
        → llama.cpp            per-token hidden states after the final norm, [n_tokens, 4096]
        → JointSchemaHead      (torch, fp16 on the GPU) → one logit per option → softmax

Hidden states from llama.cpp. Tested with llama.cpp 8212c78. Do not use embeddings = true: at this commit the output buffer then also reserves logits, n_vocab floats per token, which is about 1 MB of pinned host RAM per token. Use the nextn output instead (src/llama-ext.h). For Qwen3.5 it is output_norm(h) for every token, independent of batch.logits. Mark only the last token of each chunk as an output, so the LM head runs for one token per chunk:

cp.embeddings = false; cp.n_batch = cp.n_ubatch = 512;
llama_context * ctx = llama_init_from_model(model, cp);
llama_set_embeddings_nextn(ctx, true, /*masked=*/false);
// per request: llama_memory_clear(...); then decode the prompt in chunks of n_ubatch tokens (positions continue),
// logits flag only on each chunk's last token, and copy llama_get_embeddings_nextn(ctx) (m × 4096 floats) after each chunk

A request is computed from scratch every time (llama_memory_clear), so no state carries over between requests. Answers for the same request are bit-identical whatever ran before.

Head. Turing has no bf16, so run JointSchemaHead in fp16 (on our data, fp16 and fp32 give the same decisions as bf16). The head's output_embedding_weight argument only needs weight[token_ids]. Gather those rows from the memory-mapped lm_head_f16.safetensors and move only them to the GPU.

Native support in llama.cpp. The open llama.cpp PR #29831 adds Clef as a native architecture (clef) with a /v1/systemone endpoint in llama-server. Its GGUFs (ggml-org/Clef-Flash-GGUF) are a different format from the ones here. The files in this repository work with released llama.cpp.

Quantization

• Source: the original bf16 weights, converted to a bf16 GGUF with llama.cpp's convert_hf_to_gguf.py (--no-mtp, with Qwen3_5ForConditionalGeneration mapped to the text model).
• Base type: Q5_K_M, with these per-tensor overrides:
  • ssm_alpha, ssm_beta (gated delta net gates): F32, unquantized.
  • ssm_out and the full-attention attn_q/attn_k/attn_v/attn_output: Q8_0. The exception is ssm_out in the middle linear-attention layers, which is Q6_K.
  • Extra bits in the first and last layers (ffn_up 0–7, attn_gate, attn_qkv, ffn_down 0/27/31).
  • token_embd: Q8_0. llama.cpp keeps it in host RAM, so it costs no VRAM.
  • output: Q2_K. Unused, see above.
• Importance matrix: 1.5M tokens at context 2048, every one of them rendered in Clef's own input format (system prompt, STATE, SCHEMA FIELDS, options as JSON). Sources:
  • 5,018 decision records with 1–3 questions of all three types (noul, choice, score) in English and Czech. States come from the train splits of BoolQ, AG News and SST-2, from multi-article AG News digests, and from Czech news (CIIRC-NLP/czech_news_simple-cs, hynky/czech_news_dataset_v2).
  • bartowski's calibration_datav3, used as states.
• Disjoint from the evaluation: none of the evaluation items below are in the calibration data. The evaluation uses the validation and test splits.

Measured quality

How we measured. Each quantization ran through the full pipeline above, with the head in fp16 in every run, and we compared its option probabilities with the same pipeline on the bf16 GGUF. The items:

• 900 questions: 300 each from BoolQ (validation), AG News (test) and SST-2 (validation).
• 24 long synthetic states (2k–12k tokens) with 5 questions each.

KL is mean KL(bf16 ‖ quant) per question; agreement is the share of questions where both pick the same option. Measured on an NVIDIA GB10 with llama.cpp aa885f0:

For 0.3 GB more VRAM than a standard Q4_K_M, this build is 3.5× closer to bf16 by KL than the best public Q4_K_M we tested, and 2.2× closer than Q5_K_S, which needs more VRAM. Most of the gain comes from two things. The first is the per-tensor layout of current llama.cpp: F32 gated-delta-net gates, Q8_0 ssm_out and attention. The llama.cpp we started with applied none of it, and with it alone our Q4_K_M went from KL 0.0053 to 0.0026. The second is the Q5 backbone that the Q2_K LM head pays for. Using our decision-format imatrix instead of bartowski's general one lowers KL by a further third at the same recipe.

Running on an 8 GB GPU (measured in production)

clef-flash-Q5_K_M-dyn.gguf on one Quadro RTX 4000 8 GB (Turing, sm_75, CUDA 12.2). The serving stack:

• llama.cpp 8212c78 with the nextn bridge described above, ctx 16384, ubatch 512, flash attention, the GGUF read with O_DIRECT (load_mode = dio).
• The head in torch 2.5.1 fp16, with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

Median latency of one request, one at a time. The backbone is 98 % of it; tokenizer and head stay under 45 ms even at 16k tokens:

Requests are serialized on the one backbone. With the earlier Q4_K_M build, at the same speed, throughput stayed at ~55 requests/min for ~1k-token requests with 1, 2 or 4 parallel clients.

How it compares with the other decision models we serve

We run three decision models behind the same /v1/systemone API (Rune and Clef-Flash in production, Lux-9B in testing) and publish all three: Livesport/rune-26b-a4b-v3-GGUF, Livesport/clef-flash-GGUF and Livesport/Decision-2.0-Lux-9B-GGUF. This section is the same in all three cards.

How we measured. 7,580 questions from the validation and test splits of 12 public datasets, English and Czech, sampled with fixed seeds. Every item is one /v1/systemone request with one question; the state and the question text are the same for all three models. Each model ran from the exact file published in its repository, served as its card describes (Rune: llama-server with our adapter, request-scoped prefix reuse; Clef and Lux: the hidden-state bridge), one request at a time on an NVIDIA GB10. Accuracy is the share of questions where the most probable option is the gold label (for the star rating: the exact number of stars).

Reading the table.

• Overall the three are within half a point of each other; they differ by task.
• Rune is best on adversarial reasoning (ANLI, by 12–19 points), on Czech fact checking (CsFEVER) and Czech Facebook sentiment, and it is the best calibrated: its probabilities match observed accuracy most closely.
• Clef-Flash is best on English topic, yes/no-over-a-passage and intent tasks, and on CTKFacts.
• Lux-9B is best on MNLI, the Czech star rating and Czech product reviews, and is close to Clef on the English classification tasks.
• We do not know the training data of the three models in full. Decision models are commonly trained on public NLI, QA and intent data, so results on MNLI, BoolQ or CLINC may be partly in-distribution for some of them.
• On our own (unpublished) English and Czech decision tasks the picture was the same: Rune leads on Czech faithfulness and fact checks, Clef and Lux on English routing and matching.

Speed. Median latency of one request on an NVIDIA GB10, one request at a time:

Every server here runs one request at a time, so more parallel clients add queueing, not throughput.

On a Quadro RTX 4000 8 GB (Turing), measured in production:

• Rune IQ3_S-dyn on 2 GPUs: 0.49 s for ~730 tokens, 1.05 s for ~2.3k tokens, 4.12 s for ~8.9k tokens (1 question; 5 questions add 0.3–0.6 s).
• Clef-Flash Q5_K_M-dyn on 1 GPU: 0.57 s for ~600 tokens, 1.75 s for ~2k tokens, 6.58 s for ~7.4k tokens.
• Lux-9B Q4_K_M-dyn: not deployed on Turing yet. It has the same Qwen3.5-9B-class backbone and serving stack as Clef, so expect similar speed.
• The 9B dense backbones are about 1.6× slower than Rune's mixture of experts on Turing, but need one 8 GB GPU instead of two.

License and credits

• Clef-Flash by Cloudflare, Apache-2.0, post-trained from Qwen/Qwen3.5-9B. The joint schema head, its code, the tokenizer and the config are Cloudflare's files, unchanged.
• Quantization and serving notes: Livesport, Apache-2.0, provided as is.
• Thanks: bartowski for public quants and calibration data to compare against.