clef-omni-fp8

FP8 (E4M3, 128x128 block scales, Qwen/DeepSeek FineGrainedFP8 layout) quantization of Cloudflare/clef-omni (revision 0db1cd2). Only the thinker's MoE experts are FP8; attention, router, embeddings, lm_head, the vision and audio towers and the Clef joint decision head stay BF16.

Status: Agrees with BF16 on 95.7% of held-out decisions with TV 0.027, at the measurement's noise floor (BF16 vs itself: 95.7%, TV 0.028), so it is indistinguishable from BF16 here. Decision Index -0.20 points vs BF16 (40/42 benchmarks within 2 points). 32 GiB on the card, about 0.6 s per decision with packed FP8 experts.

Which Clef-Omni quant should I use?

Recommended for:

• clef-omni-int4: any recent GPU when you want the smallest VRAM footprint today; loads in plain transformers. Also the best bet on AMD (ROCm): AutoRound's PyTorch/Triton kernels run there, though we have not tested it.
• clef-omni-nvfp4a16: Blackwell, or vLLM's Marlin MoE path on older cards; NVFP4 weights with 16-bit activations (higher fidelity than W4A4).
• clef-omni-nvfp4: Blackwell engines with native FP4 tensor cores (W4A4 is the only variant that can run faster than BF16 there).
• clef-omni (Cloudflare BF16): the reference, if you have ~70 GB of VRAM.

Size on disk is the download. VRAM (weights) is what serve_clef.py holds on the GPU: the NVFP4 checkpoints are unpacked to BF16 there (transformers has no packed-NVFP4 kernels), so today only INT4 actually saves VRAM, and the NVFP4 files are for engines with native FP4. Agrees with BF16 is the share of 258 held-out decision questions where the quant picks the same top answer as Cloudflare's BF16 model run through the same code. BF16 against itself under a different expert kernel scores 95.7%, so about 96% is the ceiling. Decision Index is the chance-corrected public index from the Decision Index kit, edition 0.2.1 (the edition behind Cloudflare's published Clef-Omni results), on our hardware.

Quick start

Serve it with serve_clef.py (also in the repo it is maintained in), a single-file PEP 723 script, so uv installs everything it needs:

# uv runs a PEP 723 script straight from its URL (or download serve_clef.py and `uv run` it locally)
uv run https://huggingface.co/dbirks/clef-omni-fp8/resolve/main/serve_clef.py --model dbirks/clef-omni-fp8 --demo
uv run https://huggingface.co/dbirks/clef-omni-fp8/resolve/main/serve_clef.py --model dbirks/clef-omni-fp8 --port 8000

The first command answers one built-in decision and exits; the second starts the SystemOne HTTP server (GET /healthz turns 200 once the model is loaded):

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "clef-omni",
  "state": {"observation": "The player is facing a wall."},
  "questions": {"action": {"type": "choice", "criteria": {"left": "Turn left", "right": "Turn right", "forward": "Move forward"}}}
}'

Answers come from Cloudflare's own systemone() in joint_schema_model.py: one forward pass, per-option probabilities from the original BF16 joint decision head. Inputs can include images, audio and video (see the base model card).

How this checkpoint runs

serve_clef.py keeps the FP8 experts packed in VRAM (FP8 weights plus FP32 tile scales) and dequantizes each layer just in time for transformers' grouped matmul, so the head runs unchanged. transformers 5.10.2 also loads the checkpoint itself (quant_method: fp8, dequantized or native FP8 experts), and the format is the one vLLM and SGLang load for Qwen FP8 releases.

What was quantized

Recipe

scripts/clef-omni/quant_fp8_block.py: converts the per-expert BF16 tensors straight from the release shards (no model load, no calibration), using transformers 5.10.2's own block-FP8 math (Fp8Quantize: scale = 448/amax per 128x128 tile, round-to-nearest cast), and writes weight (F8_E4M3) plus weight_scale_inv (F32). Everything else is copied bit-for-bit; talker/code2wav are dropped. CPU-only, streaming, under 1.5 GiB of host RAM.

Evaluation

Parity: 128 held-out records (258 questions; text from ultrachat_200k test_sft in Clef's state/questions schema, about 20% with synthetic image/video/audio, never used for calibration). Each exported quant is loaded through serve_clef.py and compared with Cloudflare's BF16 model run through the same code path (grouped experts). We report top-answer agreement and total variation (TV) between the per-option distributions. Noise floor: the same BF16 weights run through a different but equally valid expert kernel (eager vs grouped) agree on only 95.7% of these questions (TV 0.028), because many are near-ties. Agreement around 96% and TV near 0.03 therefore means indistinguishable from BF16 at this precision. Benchmarks: the Decision Index reproduction kit, edition 0.2.1, over /v1/systemone on one RTX PRO 6000 Blackwell.

Benchmarks

Decision Index 0.2.1 (the edition behind Cloudflare's published Clef-Omni table), run with the public decision-index kit (suite rebuilt and hash-verified against the lab's manifest) over /v1/systemone on one RTX PRO 6000 Blackwell. We score a deterministic stratified sample of 3,000 requests (about 58 per benchmark, suite sample --n 3000) rather than all ~150K, so each benchmark carries roughly ±5-7 points of sampling noise; the averages and the paired deltas (both models on the identical questions, score --reference) are the reliable signals. The kit's headline index is coverage-adjusted over the full suite, so it is not meaningful on a sample and is not reported.

Version history

Each version is an annotated git tag on this repo (revision="v0.1" etc. pins it). Newest last.

License

Apache-2.0, as the base model. All credit for the model goes to Cloudflare; this repo only changes the weight precision.