Clover-35B-A3B-Preview GGUF (IQ4_XS, for Mac)

This is a 4-bit GGUF quant of Diffuspace/Clover-35B-A3B-Preview. We sized it to run on Apple Silicon Macs with 32 GB of memory. All credit for the model goes to Diffuspace. The Apache-2.0 license carries over unchanged.

Download

File	Type	Size
`clover-35b-a3b-iq4_xs-pinq8.gguf`	IQ4_XS with sensitive tensors pinned to Q8_0, 4.39 bpw	19.0 GB (19,019,892,192 bytes)	Recommended
`clover-35b-a3b-iq4_xs.gguf`	IQ4_XS, stock rules, 4.37 bpw	18.9 GB (18,942,272,992 bytes)	Previous (unpinned)

hf download austindixson/Clover-35B-A3B-Preview-GGUF-IQ4_XS-Mac clover-35b-a3b-iq4_xs-pinq8.gguf --local-dir .

SHA-256 of the pinned file: ad2a26f1bf9fa2c4d249ade6773e7129bf0eee59c94873b9e121c1c695da394b

What changed in the pinned file

Thanks to @letsgetboosted for the suggestion. Stock quant rules squash the shared-expert FFN (ffn_*_shexp) and the tiny ssm_alpha / ssm_beta tensors to 4 bits. The pinned file keeps those, plus attn_k and attn_v, at Q8_0. Everything else is the same IQ4_XS as before.

Tensor	Previous (unpinned)	Pinned
`ffn_gate_shexp`, `ffn_up_shexp`	IQ4_XS	Q8_0
`ffn_down_shexp`	IQ4_XS (35 layers), Q5_K (5 layers)	Q8_0
`ssm_alpha`, `ssm_beta`	IQ4_XS	Q8_0
`attn_k`	IQ4_XS	Q8_0
`attn_v`	Q5_K	Q8_0
Routed experts, `attn_q/gate/output`, `ssm_out`, `token_embd`	IQ4_XS (some Q5_K)	unchanged
`output` / routers / norms	Q6_K / F32 / F32	unchanged

Measured against the Q8_0 GGUF as reference with llama-perplexity --kl-divergence (40 chunks × 512 ctx on a mixed prose, code and math text, CPU backend for all three):

	Previous (unpinned)	Pinned	Change
Mean KLD vs Q8_0	0.0959 ± 0.0021	0.0861 ± 0.0023	−10.2%
99th-percentile KLD	0.757	0.698	−7.8%
Top-1 token agreement with Q8_0	88.6%	89.6%	+1.0 pp
PPL penalty vs Q8_0 (PPL(Q)/PPL(Q8_0) − 1)	3.52%	2.84%	−0.68 pp
File size	18,942,272,992 B	19,019,892,192 B	+74 MiB (+0.41%)
pp512 / tg128, llama-bench, M1 Pro 32 GB	292.2 / 22.7 tok/s	288.9 / 22.5 tok/s	within noise
Server RSS, loaded	18.39 GiB	18.38 GiB	same

How it was made

No imatrix and no calibration data were used. Both files are a Q8_0-sourced requant with stock llama.cpp quant rules; the pinned file adds only the --tensor-type overrides below.

convert_hf_to_gguf.py --remote streamed the BF16 safetensors from the Hub, so no BF16 copy was ever written to disk. Its output was Q8_0.
llama-quantize --allow-requantize took the Q8_0 down to IQ4_XS.

Pinned file:

llama-quantize --allow-requantize \
  --tensor-type ffn_gate_shexp=q8_0 --tensor-type ffn_up_shexp=q8_0 --tensor-type ffn_down_shexp=q8_0 \
  --tensor-type ssm_alpha=q8_0 --tensor-type ssm_beta=q8_0 --tensor-type attn_k=q8_0 --tensor-type attn_v=q8_0 \
  clover-35b-a3b-q8_0.gguf clover-35b-a3b-iq4_xs-pinq8.gguf IQ4_XS 8

Previous (unpinned) file: the same command without the --tensor-type flags.

Because the source was Q8_0 and not BF16, the pinned tensors are copied from Q8_0 verbatim, and quality may be slightly lower than a direct BF16 → IQ4_XS quant.

You need a patched llama.cpp

Mainline llama.cpp can't load Clover yet. Support comes from llama.cpp PR #31306 (Maphist0's dev-clover branch of llama.cpp-pub). We built from commit 4838ffb with Metal.

git clone -b dev-clover https://github.com/Maphist0/llama.cpp-pub.git llama.cpp
cd llama.cpp
git checkout 4838ffb
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server llama-bench

Chat template

These GGUFs have no embedded chat template. For chat and especially tool calling, pass Diffuspace's template explicitly:

curl -LO https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview/resolve/main/chat_template.jinja

Run it:

# chat in the terminal
./build/bin/llama-cli -m clover-35b-a3b-iq4_xs-pinq8.gguf -ngl 99 -fa 1 --repeat-penalty 1.0 --jinja --chat-template-file chat_template.jinja

# OpenAI-compatible server on localhost (tool calling works with the template)
./build/bin/llama-server -m clover-35b-a3b-iq4_xs-pinq8.gguf -ngl 99 -fa 1 --repeat-penalty 1.0 --cache-ram 0 --jinja --chat-template-file chat_template.jinja --host 127.0.0.1 --port 8080

For the previous file, swap in clover-35b-a3b-iq4_xs.gguf.

Measured speed

These numbers are from an M1 Pro with 32 GB, using the previous (unpinned) file, llama-bench -ngl 99 -fa 1 -r 2 and plain autoregressive decoding. The pinned file measured the same within noise (see the table above).

Test	Result
pp512 (prompt processing)	261 tok/s
tg512 (generation)	21.4 tok/s
Max RSS	~16.4 GB

More results on the same M1 Pro 32 GB

These were measured on the previous (unpinned) file.

In real runs, plain decoding generated about 21.7 tok/s and processed prompts at about 212–261 tok/s.
Resident memory was about 19.4 GB in a running session. On a 32 GB Mac, pass --cache-ram 0 so it doesn't swap.
Draft-self speculative mode ran 3–34% slower than plain decoding on Metal: 20.97 tok/s with exact N=7 and 14.3 tok/s with N=31. On a Mac, stick with plain decoding.
The lossy speculative mode needs --repeat-penalty 1.0. The GGUF's default of 1.05 breaks it.
On a 30-item hard MATH subset (25k-token budget, greedy) it scored 28/30 strict. Both misses were answer formatting, so it was 30/30 if you're lenient about format. The Wilson 95% CI on 28/30 is 78.7–98.2%. That's a small sample, so treat it as a spot check rather than a benchmark.

austindixson/Clover-35B-A3B-Preview-GGUF-IQ4_XS-Mac