Agnes-3.0-Flash Preview - GGUF quantizations

GGUF quantizations of Agnes-AI/Agnes-3.0-Flash (the open-weight Preview checkpoint, Apache-2.0), quantized with imatrix-weighted llama-quantize.

These files need a patched build. Agnes has an extra parallel SwiGLU branch in every layer (ffn_gate_par / ffn_up_par / ffn_down_par) that stock llama.cpp does not know about. The fork cafe-llama.cpp adds those tensors to the qwen35 architecture (plus the recurrent/full attention layer map and the delta-attention V-head reorder). Loading these GGUFs on an unpatched build will fail or mis-load.

Main ladder

Oficial llama.cpp is not supporting this yet, you have to use cafe-llama.cpp https://github.com/quimmedes/cafe-llama.cpp

Total: 212.9 GiB - all of it uploaded.

Other quantizations (extra/)

Total: 105.2 GiB. Same model and same runtime as the ladder above; different tensor-type allocation (more precision kept in attention, more compression in the FFN).

How to run

# 24 GB card, long context, MTP speculative decoding
llama-server -m Agnes-3.0-Flash-Q4.5-v4-XYZ.gguf \
  --host 127.0.0.1 --port 8080 -ngl 20 -nr -c 32768 -np 1 \
  -ctk q8_0 -ctv q8_0 -fa on --spec-type draft-mtp

• -nr (no repack) keeps weights memory-mapped, which matters at 33B.
• On a 24 GB card -ngl around 20 fits; a full offload only leaves room for a small context.
• The MTP block is included (nextn, 1 layer), so --spec-type draft-mtp works.
• Base card's recommended sampling: temp 1.0, top_p 0.95, top_k 20.

What this checkpoint actually is (findings, with the method)

While converting this model I compared it against Qwen/Qwen3.8-27B, because both report the same architecture (qwen35). The bytes say the two are the same blueprint. Method: sha256 over HTTPS for the small files, and byte-comparison of tensor ranges (safetensors data_offsets header, Range: requests) for the weights.

Byte-identical:

Different:

The parameter count agrees: 27B x 72/64 = 30.4B, plus the parallel branch (3 x 2048 x 5120 x 72 ~ 2.3B) = ~32.6B, and this GGUF measures 32.63B. Nothing is unaccounted for.

So: Agnes-3.0-Flash Preview carries the Qwen3.8-27B blueprint - same tokenizer, same chat template, same vision tower (identical weights), same embeddings/norms/SSM constants - with 8 extra delta layers, a new parallel-FFN branch, and retrained projections. Both models are Apache-2.0, so this is entirely legitimate; the reason to write it down is that people know what they are downloading.

How these were made

1. Safetensors -> GGUF with the fork's converter. Changes needed for this model: register the parallel-FFN tensors, record the delta-attention layers as recurrent (otherwise every layer looks like full attention and attn_qkv fails), and map .delta_attn./.global_attn. onto the standard qwen35 names.
2. Imatrix built from 50 chunks of the wikitext-2 raw test split; the bf16 GGUF scores PPL 6.5737 +/- 0.14183 with it.
3. Quantized with llama-quantize --imatrix, per-tensor types: attention and SSM tensors stay at higher precision, the FFN carries the compression. The parallel FFN follows the FFN of its block.

Practical consequences

• It is dense, not a MoE. No experts: 0 of the 1521 checkpoint tensors contain "expert", and the config has no MoE keys (intermediate_size 17408 plus a parallel 2048 SwiGLU). All ~33B parameters are read for every token, so expect dense-model speed at equal quantization, not "3B active" speed. The base card's benchmark table compares against models labelled "X active"; Agnes is not one.
• Long context is comparatively cheap. Only 18 of the 72 layers hold a KV cache that grows with context (the 54 delta-rule layers keep a fixed-size recurrent state). At 128k with q8_0 KV that is about 5 GB (18 x 2 x 4 KV heads x 256 dim x 131072 x 1.06 B), which is why a 24 GB card can hold the Q4.5 weights and a long context.
• Vision is not in these files. The text tower and the MTP layer are; the projector is separate. Because the vision tower weights are byte-identical to Qwen3.8-27B's, the mmproj-Qwen3.8-27B-bf16.gguf from quimmedes/Qwen3.8-27B-XYZ is expected to work - but a full image encode could not be run on this machine (the vision graph needed more RAM than was free), so treat that as unverified.
• These are quantizations of a preview checkpoint; the production Agnes model has a different configuration and its benchmarks do not apply here.

Credits

• Base model: Agnes-AI/Agnes-3.0-Flash (Apache-2.0).
• Runtime: quimmedes/cafe-llama.cpp - required for the parallel-FFN tensors.