Clef-Flash mixed-precision GGUF with vision (measured-KL)

Mixed precision here means the 3.30 GB text backbone. Tensors use IQ2_XXS, IQ2_S, Q2_K, IQ3_XXS, Q3_K, Q4_K and Q8_0; there is no ternary format (TQ1_0/TQ2_0). The vision file mmproj-Cloudflare_clef-flash-f16.gguf is 0.92 GB f16 from bartowski and was not compressed in this recipe. Total download is about 4.2 GB (4,218,158,784 bytes).

Version v1.0 (2026-10-09). Unofficial post-training quantization of Cloudflare/clef-flash revision 17f0b0ad64efb65d273590632833508766b2aae6. This is not an official Cloudflare release and is not endorsed by Cloudflare or the Qwen team. Licensed Apache-2.0 like the original (LICENSE). NOTICE.md lists the changes.

llama.cpp by itself does not produce Clef decisions: the joint schema head runs in Python on the backbone's final hidden states, and the option rows come from the original bf16 lm_head. The GGUF output.weight is Q2_K and the head does not read it.

The text GGUF general.architecture is qwen35. The llama.cpp clef graph does not export t_h_nextn, which is the hidden state this recipe reads, so the file is the Qwen3.5 text model rather than architecture clef. general.name is Snap Text because the conversion directory had that name. The weights are Clef-Flash text. Image input is a separate mmproj (architecture clip, projector qwen3vl_merger, clip.vision.projection_dim 4096).

Text-only weights are the same file as Jakevin/clef-flash-mixed-GGUF. This repo adds the mmproj and an image-capable hsdump / run_clef_gguf.py.

Size

Scores (text)

Author's private frozen development splits (kev). One seed. No confidence interval was computed. These figures are the text-backbone scores from Jakevin/clef-flash-mixed-GGUF.

Retention is this model's clean accuracy divided by the bf16 clean accuracy on the same suite.

decision-v7 can be optimistic. The imatrix and the KL allocation both used decision-v7 records: 128 calibration records, seed 1234, for the imatrix, and the first 32 of that draw (10,383 tokens) for the per-tensor KL. transfer-v9 was not used for calibration.

These are not the public Decision Index or Typesafe numbers.

Text consistency with the text-only GGUF: decision-v7 development, first 100 records, same bf16 lm_head rows. 100/100 matched rows, max |Δp| 0, 0 argmax flips. The image-capable hsdump text path is the original llama_decode loop.

Vision check

Image reading is confirmed (6 synthetic-image wiring tests, consistent with the MLX vision build). Those cases are solid red/green/blue/yellow 224×224 fields plus a blue circle and a blue square on white, compared with Jakevin/clef-flash-ternary-vision-mlx. This is a wiring check. Real-image accuracy has not been evaluated. There is no labelled image suite in this release, and original-bf16 vision accuracy was not run.

A 224×224 image through the Cloudflare/clef-flash processor expands to 64 <|image_pad|> tokens (id 248056), image_grid_thw [1, 16, 16] (upscale to 256×256; patch 16 and spatial merge 2). llama.cpp's qwen3vl default of 8..4096 tokens leaves a 224 image at 49 tokens. hsdump sets image_min_tokens=64 and image_max_tokens=16384 so the pad count matches the Hugging Face processor. A mismatch aborts.

Method

Per-tensor measured-KL sensitivity, then a knapsack that assigns each measured tensor a ggml type from IQ2_XXS, IQ2_S, Q2_K, IQ3_XXS, Q3_K, Q4_K, and Q8_0. The shipped text file is llama-quantize --tensor-type with that imatrix. alloc/selection.json is the knapsack result. alloc/tensor-types.txt is the pattern file passed to the quantizer (it pins ssm_alpha and ssm_beta to bf16; the other assignments are exact tensor names).

Counts in the text GGUF (427 tensors):

Norms, conv, ssm_alpha, and ssm_beta are not quantized. In the file, norms and conv are F32; ssm_alpha and ssm_beta are BF16. output.weight is fixed at Q2_K and is not read by the classification head. token_embd is IQ3_XXS (IQ2_XXS asserts an imatrix, and llama-quantize does not pass one for the embedding).

Sum of the per-tensor KL values used by the knapsack (not a model KL): 0.009015. Dry-run file estimate was 3.300 GB.

The mmproj has 334 tensors, f16, no deepstack layers. llama.cpp's ClefVisionModel converter raises NotImplementedError (needs ggml-org/llama.cpp#29622). The file is the Qwen3-VL converter path (Qwen3_5ForConditionalGeneration / Qwen3VLVisionModel), which is what the Clef-Flash checkpoint already is when joint_head_config.json is absent.

Comparison

Different formats and different calibration. This card does not say which allocation is better. Only numbers from the author's result files are shown; anything not measured says so.

Sizes exclude the 0.24 GB joint head, which every row needs separately. ¹ Text scores of the T-cloq16 backbone, measured on the text package; the vision package was not re-scored on the text suites. "6/6 synthetic" is the wiring check below (argmax agreement between this GGUF and the MLX vision package on the same 6 images), not an accuracy figure.

• Each number is a single run on one seed. No confidence intervals are given for any row. (The MLX v2.0 card has a paired-bootstrap CI for its v2.0 − v1.0 difference only; no GGUF-vs-MLX difference was tested.)
• This GGUF is scored with the original bf16 lm_head rows. The MLX rows use the 4-bit lm_head copy that ships in those packages (on the bf16 release the two give the same accuracy, 0.8861 / 0.8011 [a]).
• KL(bf16‖model) over all question rows was measured for the MLX builds only. For this GGUF it was not measured; the 0.009015 in Method is a sum of per-tensor KLs against a Q8_0 reference, not a model KL. Per-row probability difference vs bf16 was recorded: mean |Δp| 0.0280 (decision-v7, 32 argmax flips in 1,468 rows) and 0.0622 (transfer-v9, 96 flips in 1,264 rows) (clef-gguf-mkl-20261008/out/mkl-*/summary.json).
• Q8_0 GGUF (the per-tensor KL reference, not released) was not scored on the full suites. Only a path check on the first 100 decision-v7 dev records was run: accuracy 0.93, max |Δp| 0.0753, 1 argmax flip vs the bf16 rows [b]. (The mixed GGUF on the same 100 records: accuracy 0.93 [e].)
• Static quants such as bartowski's were not scored. hard-v1 was not run on this GGUF.

Sources (author's run folders, kev evals, development splits): [a] ternary-clef-flash-20261006/RESULTS.md (bf16 baseline and MLX v1.0 T-cloq16 evals; sizes are packed files, head excluded). [b] clef-gguf-mkl-20261008/RESULTS.md (this GGUF, full decision-v7 and transfer-v9 dev evals). [c] clef-v2-release-20261008/RESULTS.md (MLX v2.0 b31 evals and KL). [d] clef-gguf-release-20261009/RESULTS.md (bartowski file size only; not scored). [e] clef-gguf-vision-20261009/RESULTS.md (100-record text consistency check and 6-case synthetic image check).

Related repos

Listed side by side. Different formats and different calibration. This card does not say which allocation is better.

bartowski's text GGUF is architecture clef. This backbone is qwen35. The mmproj is architecture clip and pairs with this file via embedding size 4096, not via the clef graph.

Use

The joint head needs torch, safetensors, numpy, transformers, and (for images) Pillow. Classification must run in Python. hsdump needs llama.cpp b11407 or later with qwen35 and clip_graph_qwen3vl, plus libmtmd. Build it from hsdump.cpp in this folder. The two embedding calls are already in that llama.cpp (src/llama-ext.h); they are not in the installed llama.h, so the cpp file declares them. No llama.cpp patch is required at b11407 or later.

output.weight inside the GGUF is not the classification head. Pass --base as a checkout of Cloudflare/clef-flash (the same revision as above). Only lm_head.weight is read from it. The processor in that snapshot is used for image placeholders.

c++ -std=c++17 -O2 \
  -I llama.cpp/include -I llama.cpp/ggml/include -I llama.cpp/tools/mtmd \
  hsdump.cpp -o hsdump \
  -L /path/to/llama.cpp/build/bin -lllama -lmtmd \
  -Wl,-rpath,/path/to/llama.cpp/build/bin

# text
python run_clef_gguf.py \
  --base /path/to/Cloudflare/clef-flash \
  --hsdump ./hsdump

# image
python run_clef_gguf.py \
  --base /path/to/Cloudflare/clef-flash \
  --hsdump ./hsdump \
  --mmproj ./mmproj-Cloudflare_clef-flash-f16.gguf \
  --image photo.png

--model defaults to ./clef-flash-mkl-3.30GB.gguf beside the script. --hsdump defaults to ./hsdump. --image is repeatable. --record JSON may include images as paths. Video is not wired.

Text-only records use the original llama_decode path. Image records encode each RGB image with libmtmd, splice embeddings at <|image_pad|> (token 248056), and decode mixed token/embedding batches with M-RoPE positions. hsdump exits if the <|image_pad|> count disagrees with the mmproj token count.

The default record is the invoice example from the Cloudflare card (one choice question and one noul question). Hidden states are llama.cpp final-norm embeddings (llama_set_embeddings_nextn, masked false), the read used by Livesport/clef-flash-GGUF.

Limitations

• llama.cpp does not run the classifier head by itself. Clef decisions need joint_head.safetensors + joint_schema_model.py in Python (run_clef_gguf.py), hidden states from hsdump (built from hsdump.cpp against llama.cpp b11407+), and the bf16 lm_head option rows from a Cloudflare/clef-flash checkout (--base). llama.cpp chat or completion on this file is not a Clef decision, and the GGUF output.weight (Q2_K) is not the head.
• Calibration data. The imatrix used 128 records from the decision-v7 calibration split of the author's private kev suite (seed 1234, 44,619 tokens); the per-tensor KL used the first 32 of them (10,383 tokens) against a Q8_0 reference. These are Clef schema classification prompts from the same suite as decision-v7 (its dev split draws on AG News, DBpedia-14, Banking77, TREC, SST-5, IMDB, Yelp, Amazon, MNLI, BoolQ, plus compositional and policy items). decision-v7 can therefore be optimistic; transfer-v9 was not used for calibration. Other domains and languages were not checked.
• Mixed precision, not ternary. Per-tensor IQ2_XXS / IQ2_S / Q2_K / IQ3_XXS / Q3_K / Q4_K / Q8_0, with BF16/F32 for norms, conv and ssm_alpha/beta. No TQ1_0/TQ2_0. "ternary" in the related MLX repo names does not describe this file.
• Post-training quantization. No recovery training.
• No confidence intervals. Single run, one calibration seed; no paired bootstrap against bf16 or against the MLX builds.
• Evaluated sets only. Scores cover the decision-v7 (n=1264 clean) and transfer-v9 (n=1046 clean) development splits. Not run: hard-v1, the public Decision Index / Typesafe benchmarks, a model-level KL vs bf16, full-suite Q8_0 GGUF, and static quants (e.g. bartowski Q2_K).
• Vision verified on 6 synthetic images only: 224×224 solid red / green / blue / yellow, plus a blue circle and a blue square on white. GGUF 6/6 correct and argmax-agrees with the MLX ternary vision package (max |Δp| 0.0176). Compared with that MLX package, not with a bf16 vision forward. Real-image accuracy has not been evaluated.
• 224×224 → 64 image tokens. The HF processor upscales to 256×256 (patch 16, merge 2) and emits 64 <|image_pad|> tokens; llama.cpp's qwen3vl default (8..4096) would give 49. hsdump forces image_min_tokens=64 / image_max_tokens=16384; if the counts disagree it aborts. Only 224×224 inputs were exercised.
• The mmproj is bartowski's 0.92 GB f16 file, not compressed in this recipe.
• Video is not wired.
• Needs llama.cpp b11407+ with qwen35, clip_graph_qwen3vl, and libmtmd.

License and attribution

Derived from Cloudflare/clef-flash (© Cloudflare, Apache-2.0), itself post-trained from Qwen/Qwen3.5-9B (Apache-2.0). The mmproj is from bartowski/Cloudflare_clef-flash-GGUF (Apache-2.0, sha256 c8a6e9e00e1baf6433c4874dbc73bed2a56e887bcbb582fffcb00b533bc638dd). The hidden-state read follows Livesport/clef-flash-GGUF. Inference runs on llama.cpp.

繁中摘要

這是 Cloudflare/clef-flash 的 measured-KL 混合精度 GGUF 讀圖版。「mixed precision」只指 3.30 GB（3,299,992,800 bytes）文字主幹；視覺檔 mmproj 是 0.92 GB（918,165,984 bytes）f16，來自 bartowski，沒經過我們的壓縮。總下載約 4.2 GB。decision-v7 正確率 0.8813（bf16 0.8861，保留 99.5%），transfer-v9 0.7859（bf16 0.8011，保留 98.1%）。單一 seed，沒有信賴區間。imatrix 與 KL 用了 decision-v7 的校正資料（imatrix 128 筆、seed 1234；KL 用其中前 32 筆）， 所以 decision-v7 可能偏樂觀。讀圖版文字一致性：decision-v7 前 100 筆與文字版 100/100 一致，max |Δp| 0。已確認能讀圖（6 題合成圖接線測試，與 MLX 視覺版 一致）；真實圖片準確率尚未評測。224×224 圖會展開成 64 個 image token，所以 hsdump 傳 image_min_tokens=64。分類要跑 Python 的 JointSchemaHead，需要 llama.cpp b11407+ 與 hsdump；llama.cpp 單獨跑不出 clef 的判斷。