clef-flash-NVFP4
This is an NVFP4 (W4A4) quantization of Cloudflare/clef-flash
at revision 17f0b0ad.
- Quantized: the 96 text MLP projections (
gate/up/down_proj). - Kept in BF16: the gated-deltanet and full-attention layers,
lm_head(the joint head reads its rows), the vision tower and the joint head. - Size: 12.1 GB, versus 19.1 GB for BF16 (decimal GB).
Method
- Tool: llmcompressor 0.13.0
oneshotwithQuantizationModifier(scheme="NVFP4"). Weights use FP4 with group size 16 and FP8 scales. - Calibration: per-tensor input-activation global scales, taken from 248 records (3.4M tokens, up to 87K tokens long) built from legal, news, science, books, tools, routing and tax sources.
- Decontamination: the calibration data was checked against the evaluation sets.
- Stack: torch 2.13 (cu132), transformers 5.14.1, compressed-tensors 0.18.0, on an RTX PRO 6000 (SM120).
Parity vs BF16
The held-out set has 256 records, one question each: LongBench v2 (128, 4 options), banking77 (64, 77 options) and LEDGAR (64, 100 options). Inputs run from 1.7K to 237K tokens. Each model gets one forward pass per record through the joint head.
| BF16 | NVFP4 (this repo) | NVFP4 incl. attention + deltanet (not released) | |
|---|---|---|---|
| Accuracy | 62.1% | 63.7% | 60.9% |
| Δ accuracy, 95% CI (paired bootstrap) | +1.6 pt [−1.2, +4.3] | −1.2 pt [−5.1, +2.7] | |
| Top-1 agreement with BF16 | 0.926 [0.887, 0.952] | 0.883 [0.838, 0.917] | |
| Agreement where BF16 margin ≥ 0.5 (n=156) | 1.000 | 0.994 | |
| Mean KL(BF16 ‖ quant) | 0.020 | 0.048 | |
| Size (GB) | 19.1 | 12.1 | 9.1 |
Release gates: the accuracy delta's 95% lower bound must be ≥ −2 pt, and agreement on confident decisions must be ≥ 0.98. This model passes both. The variant that also quantizes attention and deltanet fails the accuracy gate, losing mostly at 8K–32K tokens (−6.1 pt).
Every disagreement with BF16 is on a question where BF16's own margin is below 0.5.
These gates were set after the first full report. Under the originally proposed gates (overall agreement ≥ 0.95, every length band ≥ 0.90) this model fails at 0.926 overall and 0.878 on the 8K–32K band. Those gates were dropped because band slices of n=26–53 are too small to gate on.
Per-band, per-cohort and per-margin breakdowns are in metrics/.
Usage
The repo uses the same code as the base model (joint_schema_model.py). Load the
backbone decompressed: the joint head calls model.language_model directly, so the
packed weights have to be expanded at load time.
import json, torch, joint_schema_model as jsm
from safetensors.torch import load_file
from transformers import Qwen3_5ForConditionalGeneration, CompressedTensorsConfig
d = "<local path to this repo>"
backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
d, dtype=torch.bfloat16, device_map={"": "cuda:0"},
quantization_config=CompressedTensorsConfig(run_compressed=False))
head = jsm.JointSchemaHead(**json.load(open(f"{d}/joint_head_config.json")))
head.load_state_dict(load_file(f"{d}/joint_head.safetensors"))
model = jsm.ClefModel(backbone, head.to("cuda:0", torch.bfloat16)).eval()
The metrics above were measured with this path, which simulates W4A4 numerically in eager PyTorch. Native FP4 kernel serving (e.g. vLLM on Blackwell) has not been evaluated.
License
Apache-2.0, inherited from the base model. This is a quantized derivative of Cloudflare/clef-flash. The weights were not otherwise modified.