Agens Volundr 32B v1 — GGUF

Main model card, benchmarks and discussion: Blockway/Agens-Volundr-32B-v1

GGUF files of Agens Volundr 32B v1 for llama.cpp: BF16, Q8_0 and Q4_K_M. Text only. For what the model is, its benchmarks and its chat format, see the main model card.

Needs the volundr branch of llama.cpp: github.com/BlockWayz/llama.cpp. Volundr is a new architecture (KDA linear attention, BCSA sparse attention, Engram memory, mHC hyper-connections). Stock llama.cpp, and apps built on it, will refuse these files with unknown model architecture: 'volundr' until support is merged upstream.

Files

Norms, convolution, decay and mHC static parameters are F32 in every file. The linear-attention (KDA) weights are kept at Q8_0 in the 4-bit file on purpose: an earlier model of ours degraded badly when its linear-attention projections were quantised to 4 bit. SHA256SUMS lists the checksums.

Build llama.cpp (volundr branch)

git clone --branch volundr https://github.com/BlockWayz/llama.cpp && cd llama.cpp

# CPU
cmake -B build -DCMAKE_BUILD_TYPE=Release
# or NVIDIA GPU
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON

cmake --build build -j --target llama-server llama-cli

If CMake cannot find libcurl, add -DLLAMA_CURL=OFF. More detail in README-volundr.md.

Run

pip install -U huggingface_hub
hf download Blockway/Agens-Volundr-32B-v1-GGUF Agens-Volundr-32B-v1-Q4_K_M.gguf --local-dir .

# OpenAI-compatible server on :8080 (thinking, tool calls and reasoning_content parsing work out of the box)
./build/bin/llama-server -m Agens-Volundr-32B-v1-Q4_K_M.gguf --jinja -c 32768 -ngl 99

# interactive chat in the terminal
./build/bin/llama-cli -m Agens-Volundr-32B-v1-Q4_K_M.gguf --jinja -c 32768 -ngl 99

Drop -ngl 99 to run on the CPU only. When offloading to a GPU, the four Engram tables (4.3 GB of pure lookups) can stay in host RAM: --override-tensor "engram_embd=CPU".

Thinking is on by default. Per request:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "What is 17 * 23 + 9?"}],
  "chat_template_kwargs": {"reasoning_effort": "low"}
}'
# "chat_template_kwargs": {"enable_thinking": false}      -> no thinking
# "chat_template_kwargs": {"reasoning_effort": "medium"}   -> low | medium | xhigh (default xhigh)

Tool calls use the standard OpenAI tools field and come back as tool_calls. Sampling defaults stored in the GGUF: temperature 1.0, top-k 20, top-p 0.95. The chat template is the model's own and is embedded in every file.

Multi-turn chat: prompt cache

llama-server reuses the cached state across the turns of a conversation, so only the new tokens of each turn are processed (the recurrent KDA/Engram state is rolled back through context checkpoints taken during prompt processing; nothing has to be enabled). Measured with the Q4_K_M file on one 48 GB GPU, -np 1, greedy:

Restoring a checkpoint is exact (bit-identical logits, llama-volundr-state-test). Flags: --ctx-checkpoints N (default 32, 168 MiB each in host RAM; 8 is plenty for a chat), --cache-ram (host prompt cache for idle conversations), one slot per concurrent conversation (-np). --cache-reuse (KV shifting) is not supported.

Speed

Q4_K_M on one 48 GB GPU, full offload (-ngl 99), flash attention, F16 KV cache:

(Preview measurements; the v1 files have the same tensor layout and types. In this build's checks: 26.5 tok/s at a 1.1K prompt with a 4-slot server, prompt processing 1,480 tok/s at 3K and 920 tok/s at 16K.)

Quantisation quality

llama-perplexity on held-out chat text (mostly Cantonese and Chinese), 12 chunks of 4,096 tokens; KLD and top-1 are measured against the BF16 file.

With an 8,192-token context scored on positions 4,096–8,191 (where the BCSA far field is active) the picture is the same: 3.1078 / 3.1086 / 3.1337.

Parity with the reference implementation

The llama.cpp graph was checked against the checkpoint's own PyTorch code (fp32) on all-position logits: the BF16 file agrees on 100 % of top-1 tokens (mean KL 1–2e-5) on an English prose prompt and a chat-templated prompt, and the 64-token greedy continuation is identical on the first and diverges on the second only at a 0.02-logit near-tie. Restoring a saved sequence state or a context checkpoint and re-processing the same tokens gives bit-identical logits.

Unused-token mask

The output layer has 248,320 rows but the tokenizer ends at id 248,076. The ids 248,077–248,319 are never valid output, yet the model can put real probability on some of them: where a tool call opens, sampling sometimes picks one instead of the tool-call token and the call comes back as plain text. The volundr branch therefore masks that range (a -inf logit bias, applied automatically for this architecture in llama-server and llama-cli). With the mask, sampled tool calls with the Q4_K_M file parsed 104/104 (no-think at temperature 0.6 and 1.0, thinking at 1.0).

VOLUNDR_MASK_UNUSED_TOKENS=0 turns the mask off. If you call the llama.cpp library directly (bindings that do not use llama.cpp's common sampling code), apply the same bias yourself.

Known limits

• Text only. The vision tower is not converted; use the main repository for image input.
• No speculative decoding in llama.cpp.
• CUDA and CPU only for the two Volundr-specific ops (Engram hashing, mHC Sinkhorn). On Metal, Vulkan or ROCm the scheduler should run those two ops on the CPU; those backends have not been tested.
• BCSA prefill cost. Prompt processing scores every query against every cached position before pooling and selecting blocks, so it slows down with prompt length (about 620 tok/s for a 32K-token prompt on the same GPU). Decoding switches to a sparse path above 12,288 cached positions.
• Tokenizer. llama.cpp applies no Unicode normalisation (use NFC input, as the HF tokenizer does), and for scripts with combining marks (Devanagari, Thai, Burmese, Tamil, ...) its Qwen3.5 pre-tokenizer keeps marks inside letter runs where the HF tokenizers library splits them, so those languages tokenise differently from the HF/sglang model. English, Chinese, Cantonese, Vietnamese, Japanese, Korean, Arabic and code tokenise identically.
• No context shift and no partial KV-cache removal (recurrent state), as for llama.cpp's other hybrid models.
• Multi-GPU --split-mode layer works but is no faster than one 48 GB GPU; --split-mode row does not load.

Licence

Apache-2.0. These files are a format conversion of the weights in Blockway/Agens-Volundr-32B-v1; LICENSE, NOTICE and the files they reference apply.