clef-flash-NVFP4

Unofficial NVFP4 quantization of Cloudflare/clef-flash, a 9B decision model that scores every option of every typed question (choice / score / true-false) in one forward pass. This repo is not affiliated with or endorsed by Cloudflare.

It ships with a small vLLM plugin and a Jev/SystemOne-compatible HTTP server (POST /v1/systemone), because stock vLLM cannot run Clef's custom joint-schema head. In short: on a single GPU it runs 3x faster than BF16 vLLM on two GPUs (6.5x the original transformers code), at about -0.5 points of accuracy and 94% top-1 agreement with BF16.

[!IMPORTANT] Read before using

• Text and JSON state only. The vision tower is kept in BF16 in the checkpoint but has not been tested; the server rejects images / videos with HTTP 400 and runs vLLM with the vision tower disabled.
• Use the bundled vLLM plugin. The quantized checkpoint does not run efficiently in plain transformers (it can only be dequantized to BF16, which removes the memory and speed benefit). The plugin is pinned to vLLM 0.28.x; it uses vLLM pooling internals that may change in later versions.
• Hardware: NVIDIA Blackwell (SM 10.x / 12.x: B200, RTX 50-series, RTX PRO 6000, DGX Spark). Older GPUs have no FP4 tensor cores.
• Tested on one setup only: 2x RTX 5070 Ti 16 GB (SM 12.0, PCIe, no NVLink), vLLM 0.28.0, CUDA 13.
• Evaluated on 1,400 records from 7 public tasks with our own harness (below), not on Cloudflare's Decision Index. Expect a similar but not identical picture on your data; check your own workload before relying on it.

Quickstart: SystemOne API server

hf download kurcontko/clef-flash-NVFP4 --local-dir clef-flash-NVFP4

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD/clef-flash-NVFP4:/model:ro \
  --entrypoint bash vllm/vllm-openai:v0.28.0 -c \
  "cp -r /model/vllm_plugin /tmp/p && pip install -q /tmp/p && clef-systemone --model /model --dp 1"

--dp N runs one engine per GPU behind the one port (recommended over tensor parallelism on PCIe GPUs). The server warms up its kernels before /health reports ready (about 3-4 minutes from a cold container).

curl localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "clef-flash",
  "state": "Our checkout started returning errors and orders are blocked.",
  "questions": {
    "department": {"type": "choice", "instructions": "Which team should handle the message?",
                   "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},
    "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
    "outage": {"type": "noul", "instructions": "Is a service down?"}}}'

The request and response bodies are the same as the systemone() function in the original release: answers keyed by question ID (choice with confidence and probabilities, score with the expected score and legend, noul with the probability of true) and usage. Also served: GET /v1/models, GET /health.

Quickstart: Python (offline vLLM)

# pip install ./clef-flash-NVFP4/vllm_plugin   (in an environment with vllm==0.28.*)
import sys
from transformers import AutoTokenizer
from vllm import LLM, PoolingParams
from clef_vllm import ARCHITECTURE, TASK, schema_layout

path = "./clef-flash-NVFP4"
sys.path.insert(0, path)
from joint_schema_model import encode_record


def main():
    # Keep `llm` local: vLLM shuts its engine process down when the LLM object is freed.
    llm = LLM(model=path, hf_overrides={"architectures": [ARCHITECTURE]}, runner="pooling",
              enable_prefix_caching=False, language_model_only=True, max_model_len=16384)
    tokenizer = AutoTokenizer.from_pretrained(path)
    record = {
        "state": {"invoice": {"vendor": "Acme", "total": 1250.0, "currency": "USD", "status": "overdue"}},
        "questions": {
            "status": {"type": "choice", "instructions": "What is the invoice status?",
                       "criteria": {"paid": "Invoice is paid.", "overdue": "Invoice is past due.", "draft": "Not sent."}},
            "large": {"type": "noul", "instructions": "Is the total above 1000 USD?"},
        },
    }
    encoded = encode_record(tokenizer, record)
    params = PoolingParams(task=TASK, extra_kwargs={"clef_questions": schema_layout(encoded)})
    output = llm.encode([{"prompt_token_ids": list(encoded.input_ids)}], pooling_params=[params], pooling_task=TASK)[0]

    logits, offset = output.outputs.data.float(), 0   # every option logit of every question, in order
    for question in encoded.questions:
        n = len(question.option_ids)
        print(question.question_id, dict(zip(question.option_ids, logits[offset:offset + n].softmax(-1).tolist())))
        offset += n


if __name__ == "__main__":
    main()

Pass a list of prompts and a matching list of PoolingParams to score many records in one call; vLLM batches them.

Results

Accuracy and agreement with BF16

1,400 records, 200 per task, from held-out splits: BANKING77 (test), ARC-Challenge (test), MMLU (test), HellaSwag (validation), ANLI R3 (test), BoolQ (validation), and a multi-question ANLI R1 set (3 questions per record: choice + true/false + score). Accuracy is against gold labels. Agree is top-1 agreement with the BF16 model and KL is mean KL(BF16 ‖ quantized) per question. BF16 is the original release run with transformers 5.10.2; the quantized rows were measured in vLLM with real FP8/FP4 kernels.

Accuracy differences of under 1 point are within noise at 200 records per task. The BF16 model run through the same plugin in vLLM agrees with transformers on 99.8% of questions (KL 0.0001), so the plugin itself adds no error. NVFP4 moves most on hard, near-chance cases (ANLI): treat its confidences on borderline inputs with care.

Throughput

Same 1,400 records (717k tokens, ~512 per record), RTX 5070 Ti 16 GB GPUs.

Through the HTTP server with 64 concurrent clients: NVFP4 on 1 GPU 39.7 req/s (p50 0.95 s, p95 5.1 s), NVFP4 on 2 GPUs 78.7 req/s; with 8 concurrent clients on 2 GPUs, p50 latency is 132 ms. On PCIe GPUs, tensor parallelism was slower than one engine per GPU (FP8 TP2: 8,605 tok/s; NVFP4 TP2: 10,850 tok/s).

Single-request latency

One request at a time over HTTP, end to end, on one RTX 5070 Ti with --gdn-backend flashinfer (median of 100):

Two settings in the bundled server account for this; with vLLM 0.28's defaults the same requests took about 70 ms:

• CUDA graphs sized for prompts. vLLM captures graphs only for steps of up to 2 x max_num_seqs tokens, which fits chat decoding but leaves every prefill-only Clef request on the slow eager path. The server captures graphs for steps of up to 2,048 tokens (--max-cudagraph-tokens, about 0.6 GiB of GPU memory): 70 ms to 37 ms.
• --gdn-backend flashinfer on RTX 50-series. vLLM 0.28 enables FlashInfer's kernel for Qwen3.5's linear attention only on SM 9.0 / 10.x and falls back to Triton on SM 12.0. Forcing it works on an RTX 5070 Ti with unchanged outputs (94.4% agreement with BF16): 37 ms to 34 ms, and one-GPU throughput rises from 20.4k to 22.4k tok/s. Experimental and opt-in; tested on this card only.

Responses carry a Server-Timing header (encode, engine, answer). For the lowest latency run one engine (--dp 1); --dp 2 adds a process hop of a few milliseconds.

Quantization details

• Tool: llm-compressor 0.14.0 (compressed-tensors 0.19.0), QuantizationModifier(targets="Linear", scheme="NVFP4").
• Scheme: NVFP4 W4A4: FP4 (E2M1) weights and activations, 16-element groups with FP8 (E4M3) group scales and one FP32 global scale per tensor; activation global scales calibrated, group scales dynamic.
• Calibration: Yes: 672 records (96 from the train split of each of the 7 eval tasks below, encoded exactly as at inference).
• Kept in BF16: lm_head (the joint head reads its rows as lexical option embeddings, so it is never quantized), the vision tower (model.visual.*), the Gated DeltaNet gate projections linear_attn.in_proj_a / in_proj_b (32 outputs each), embeddings, norms, and the joint schema head (clef_head/).
• Gated DeltaNet fused scale: vLLM fuses linear_attn.in_proj_qkv and in_proj_z into one GEMM, which needs a shared NVFP4 weight global scale. llm-compressor 0.14 only shares scales for q/k/v and gate/up, so scripts/quantize.py adds ("in_proj_qkv", "in_proj_z") to its fused groups. Without this, vLLM applies the larger scale to both halves and the qkv weights come out scaled by about 0.67 (we measured 92.9% instead of 94.3% agreement). Anyone re-quantizing Qwen3.5-family models to NVFP4 for vLLM should do the same.
• recipe.yaml is the exact recipe llm-compressor saved.

How the vLLM plugin works

vllm_plugin/ (pip install-able, package clef-vllm) registers ClefFlashForDecision, a subclass of vLLM's Qwen3_5ForConditionalGeneration that runs as a pooling model. Its pooler collects each request's final hidden states across chunked prefill and runs the original JointSchemaHead (from joint_schema_model.py) on the full prompt. Each request carries its schema layout (question and option token spans from encode_record) in PoolingParams.extra_kwargs["clef_questions"]; the output is one flat vector of every option logit. Prefix caching must be off. Under tensor parallelism the vocab-sharded lm_head is gathered once on rank 0 for the head's lookups.

Files

See also kurcontko/clef-flash-FP8-Dynamic (near-lossless, about half the speed).

License and attribution

Apache-2.0, following Cloudflare/clef-flash (itself post-trained from Qwen/Qwen3.5-9B). All credit for the model goes to its original authors; this repository only adds the quantization, the vLLM plugin and the server. Not an official Cloudflare release.