Clover-35B-A3B-Preview-GGUF

GGUF quantizations of Diffuspace/Clover-35B-A3B-Preview.

[!NOTE] Clover support is not in llama.cpp master. These files were made and checked with commit 4838ffb of llama.cpp PR #31306 (Maphist0:dev-clover), which was not merged at that time.

The files are evaluated only by the KL-divergence measurements below, not by task benchmarks, long-context use or speed tests.

Files

All files are Q4_K_M with up to two kinds of changes. The -hq part raises the tensors outside the routed experts to Q8_0: attention, the linear-attention (SSM) projections and the shared expert. The -exps part changes the routed expert tensors (ffn_{gate,up,down}_exps). The tensors that only Clover has (the right-side convolution and the block-softmax parameters of the linear-attention layers) are F32 in every file, as the converter writes them.

Usage

Build llama.cpp at the commit above, then run the same command as the official GGUF with one of these files. llama-speculative-simple reads a raw prompt, so include the chat format:

cat > prompt.txt <<'EOF'
<|im_start|>user
What is diffusion?<|im_end|>
<|im_start|>assistant
<think>
EOF

llama-speculative-simple \
  -m Clover-35B-A3B-Preview-IQ4_XS-exps-hq.gguf \
  -f prompt.txt \
  --spec-type draft-self \
  --spec-draft-n-max 15 \
  --spec-accept-ratio 0.9 \
  --spec-accept-delta 0.6 \
  --temp 0 --top-k 1 --seed 42 \
  -ctk bf16 -ctv bf16 \
  -c 8192 -n 4096 -b 256 -ub 256 \
  -ngl 99 -t 8 -tb 8 \
  --flash-attn on --color off

--spec-accept-ratio and --spec-accept-delta (the PR's entropy-based draft acceptance) work only with llama-speculative-simple and greedy sampling. llama-server also accepts --spec-type draft-self --spec-draft-n-max 15. Without --spec-type draft-self, the model generates one token at a time like an autoregressive model.

KL divergence

Against the BF16 conversion of the original weights, on English Wikipedia (wikitext-2 test, 8 chunks of 1024 tokens).

BF16 perplexity on this text is 12.07. "Quality" in the Files table is described by the share of tokens whose top prediction matches BF16 and by the mean KLD relative to Q8_0.

How the files were made

• Conversion: convert_hf_to_gguf.py of the commit above, --outtype bf16, from revision 4b0402b of the original repository (transformers 5.21.0, torch 2.15.0).
• Importance matrix: llama-imatrix on the Q8_0 file over 367 chunks of 512 tokens from bartowski's calibration_datav3 and TFMC's imatrix dataset. About a third of the chunks are wrapped in a Qwen-style chat format, including <think> and tool-call round trips. The KLD text is not in the calibration data.
• Quantization: llama-quantize --imatrix imatrix.gguf --tensor-type-file tensor-types-<name>.txt <BF16> <out> Q4_K_M.
• KLD: llama-perplexity --kl-divergence with -c 1024, all on the CPU.