clef-omni-nvfp4

NVFP4 (W4A4) quantization of Cloudflare/clef-omni (revision 0db1cd2) made with NVIDIA ModelOpt 0.47.0. 22.0 GB instead of 70.8 GB. Only the thinker's MoE experts (about 91% of the loaded weights) are FP4; attention, router, embeddings, lm_head, the vision and audio towers and the Clef joint decision head stay BF16.

Status: experimental (max calibration). Faithful W4A4 replay agrees with BF16 on 92.6% of held-out decisions (TV 0.059) and scores 0.76 points below BF16 on the Decision Index; the INT4 sibling is the better choice for most people.

Which Clef-Omni quant should I use?

Recommended for:

• clef-omni-int4: any recent GPU when you want the smallest VRAM footprint today; loads in plain transformers. Also the best bet on AMD (ROCm): AutoRound's PyTorch/Triton kernels run there, though we have not tested it.
• clef-omni-nvfp4: Blackwell engines with native FP4 tensor cores (W4A4 is the only variant that can run faster than BF16 there).
• clef-omni (Cloudflare BF16): the reference, if you have ~70 GB of VRAM.

Size on disk is the download. VRAM (weights) is what serve_clef.py holds on the GPU: the NVFP4 checkpoints are unpacked to BF16 there (transformers has no packed-NVFP4 kernels), so today only INT4 actually saves VRAM, and the NVFP4 files are for engines with native FP4. Agrees with BF16 is the share of 258 held-out decision questions where the quant picks the same top answer as Cloudflare's BF16 model run through the same code. BF16 against itself under a different expert kernel scores 95.7%, so about 96% is the ceiling. Decision Index is the chance-corrected public index from the Decision Index kit, edition 0.2.1 (the edition behind Cloudflare's published Clef-Omni results), on our hardware.

Quick start

Serve it with serve_clef.py (also in the repo it is maintained in), a single-file PEP 723 script, so uv installs everything it needs:

# uv runs a PEP 723 script straight from its URL (or download serve_clef.py and `uv run` it locally)
uv run https://huggingface.co/dbirks/clef-omni-nvfp4/resolve/main/serve_clef.py --model dbirks/clef-omni-nvfp4 --demo
uv run https://huggingface.co/dbirks/clef-omni-nvfp4/resolve/main/serve_clef.py --model dbirks/clef-omni-nvfp4 --port 8000

The first command answers one built-in decision and exits; the second starts the SystemOne HTTP server (GET /healthz turns 200 once the model is loaded):

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "clef-omni",
  "state": {"observation": "The player is facing a wall."},
  "questions": {"action": {"type": "choice", "criteria": {"left": "Turn left", "right": "Turn right", "forward": "Move forward"}}}
}'

Answers come from Cloudflare's own systemone() in joint_schema_model.py: one forward pass, per-option probabilities from the original BF16 joint decision head. Inputs can include images, audio and video (see the base model card).

How this checkpoint runs

serve_clef.py keeps the FP4 experts packed in VRAM and unpacks each layer just in time for transformers' grouped kernels: about 28 GB on the card (15.3 GiB of expert weights) at about 0.55 s per decision. --no-packed-nvfp4 unpacks everything at load (about 60 GB, about 0.16 s). By default activations stay BF16; --simulate-fp4-activations also rounds expert inputs to NVFP4 exactly as ModelOpt calibrated them (faithful W4A4, several seconds per decision, used for the benchmarks). Native FP4 execution needs an engine with NVFP4 kernels plus a hidden-state path into the Clef head.

What was quantized

Recipe

ModelOpt NVFP4_EXPERTS_ONLY_CFG, max calibration, plus exclusions for towers, talker, lm_head, router and embeddings; torch 2.11.0+cu130, transformers 5.10.2 (fused MoE experts, quantized per expert by ModelOpt 0.47 and split back on export); 512 Clef-format calibration records (383K tokens), 52 min on one RTX PRO 6000.

Evaluation

Parity: 128 held-out records (258 questions; text from ultrachat_200k test_sft in Clef's state/questions schema, about 20% with synthetic image/video/audio, never used for calibration). Each exported quant is loaded through serve_clef.py and compared with Cloudflare's BF16 model run through the same code path (grouped experts). We report top-answer agreement and total variation (TV) between the per-option distributions. Noise floor: the same BF16 weights run through a different but equally valid expert kernel (eager vs grouped) agree on only 95.7% of these questions (TV 0.028), because many are near-ties. Agreement around 96% and TV near 0.03 therefore means indistinguishable from BF16 at this precision. Benchmarks: the Decision Index reproduction kit, edition 0.2.1, over /v1/systemone on one RTX PRO 6000 Blackwell.

Benchmarks for this checkpoint use the faithful W4A4 replay (--simulate-fp4-activations).

Benchmarks

Decision Index 0.2.1 (the edition behind Cloudflare's published Clef-Omni table), run with the public decision-index kit (suite rebuilt and hash-verified against the lab's manifest) over /v1/systemone on one RTX PRO 6000 Blackwell. We score a deterministic stratified sample of 3,000 requests (about 58 per benchmark, suite sample --n 3000) rather than all ~150K, so each benchmark carries roughly ±5-7 points of sampling noise; the averages and the paired deltas (both models on the identical questions, score --reference) are the reliable signals. The kit's headline index is coverage-adjusted over the full suite, so it is not meaningful on a sample and is not reported.

Version history

Each version is an annotated git tag on this repo (revision="v0.1" etc. pins it). Newest last.

License

Apache-2.0, as the base model. All credit for the model goes to Cloudflare; this repo only changes the weight precision.