How to Run Arete 27B Locally

Built from Nous Research's original weights with our own importance matrix. The calibration corpora behind our builds are public.

• Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.
• See our quantization analysis below for measurements and instructions.

---

Arete-27B is Nous Research's supervised fine-tune of Qwen3.8-27B. Nous trained it on text to change how the model answers politically sensitive questions; their model card has the details and their evaluation. The architecture, tokenizer and vision encoder are Qwen3.8-27B's. One difference matters for running it: Arete ships without the multi token prediction head the base model has. Both models are Apache-2.0; the licence file is in this repo.

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

AD- marks an Atomic Dynamic layout. The name says what the two largest tensor groups got: AD-<ffn_down>-<ffn_up>, collapsed to one name when both match. Nothing is named after a type it does not contain.

Which one fits your card

The file has to fit, and so does the context. This model keeps 64 KB of attention cache per token, which is 0.5 GB at 8k context and 2.0 GB at 32k. Budget for both.

[!TIP] If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words.

Running it

llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192

The chat template is inside every file, llama.cpp applies it on its own.

Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Thinking is on by default. With thinking on, the template also puts a reasoning effort line at the top of the system message. The effort can be xhigh (the default), medium or low. Any other value, high included, makes the template stop with an error, so a client that sends reasoning_effort: high gets nothing back. To switch thinking off or shorten it:

# no thinking
llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192 --chat-template-kwargs '{"enable_thinking": false}'

# shorter thinking
llama-server -m Arete-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192 --chat-template-kwargs '{"reasoning_effort": "low"}'

There is no multi token prediction head in these files, because the released model has none. The --spec-type draft-mtp option from our Qwen3.8-27B card does not apply here.

Images and video

This model reads images, not only text. That needs one extra file, the vision projector, which is separate from the quant and shared by all of them:

• mmproj-Arete-27B-F16.gguf for most people
• mmproj-Arete-27B-BF16.gguf if your setup prefers bfloat16

Download it once alongside whichever quant you picked:

llama-mtmd-cli \
  -m Arete-27B-AD-Q4_K_M.gguf \
  --mmproj mmproj-Arete-27B-F16.gguf \
  --image your-photo.jpg --image-min-tokens 1024 \
  -ngl 99 -c 8192 \
  -p "What is in this image?"

--image-min-tokens 1024 is not optional in practice. llama.cpp warns that this family needs at least that many image tokens, or anything that depends on position becomes unreliable.

How these compare to other builds

Numbers taken against different references cannot be put in the same table, so we did not copy anyone's published figures. We downloaded bartowski's files for this model and measured them ourselves: the same original weights, the same held-out text, the same llama.cpp build, the same kind of GPU. As a check, we measured our own AD-Q4_K_M again in that run and got the same result to the last digit. Their logs are in the metrics repo next to ours.

At sizes where the comparison is direct:

Each cell gives KL divergence, then top-1 agreement.

[!NOTE] Where we are not ahead. At Q5 the two builds are within measurement noise; ours is 0.8 GB smaller, and that is the whole difference. bartowski also ships sizes we do not: Q3_K_M at 13.2 GB (0.05514) and smaller files, IQ4_XS at 15.2 GB (0.01727), Q5_K_S at 19.3 GB (0.00426) and Q6_K at 23.6 GB (0.00152). If your card needs one of those exact sizes, take theirs.

What we found while building these

The Qwen3.8-27B layout carries over

The fine-tune changed weights, not shapes, so these files use the layout we built for Qwen3.8-27B. The first 12 and last 12 layers, the attention gate and the state output get extra bits. Every tensor row divides by 256, so no type is silently swapped for another. Each file here is about 0.3 GB smaller than our Qwen3.8-27B file with the same name, because there is no prediction head to store.

Same tokenizer in llama.cpp

Arete's tokenizer.json was re-saved with a slightly different pre-tokenizer pattern. After conversion, the GGUF tokenizer is identical to Qwen3.8-27B's: same vocabulary, merges, special tokens and pre-tokenizer type. The calibration corpus rendered for Qwen3.8-27B therefore applies unchanged, chat template included.

Q8_0 is not lossless

It is very close, but it is not the original: 0.00069 divergence and 98.87% top-1 agreement. Every number above is measured against the real BF16 weights, not against Q8_0.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe qwen3.8-27b. It is 4,967,044 tokens across 3,004 documents:

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

[!IMPORTANT] If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 4.5592 plus or minus 0.02428.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 87 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 1x H100 PCIe.

The reference logits take 88 GB, so they are not kept in the metrics repo. The manifest of the file we used is there (kld/base-neutral.manifest.txt). To measure your own build against the same point, rebuild the reference from the original weights and the held-out text (eval/ in the calibration dataset), then compare:

llama-perplexity -m Arete-27B-BF16.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld -c 4096 -ngl 99

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, in one run:

llama-imatrix -m Arete-27B-BF16.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special

Then, for AD-Q4_K_M:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.(0|1|2|3|52|53|54|55|56|57|58|59|60|61|62|63)\.ffn_(down|gate|up)\.weight$=q5_k' \
  --tensor-type '^blk\.(4|5|6|7|8|9|10|11)\.ffn_(down|gate|up)\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_down\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.ffn_gate\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.ffn_up\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.attn_(q|qkv)\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.attn_gate\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ssm_out\.weight$=q5_k' \
  --tensor-type '^output\.weight$=q6_k' \
  --tensor-type '^blk\.\d+\.attn_output\.weight$=q6_k' \
  --tensor-type '^token_embd\.weight$=iq4_xs' \
  --override-kv general.file_type=int:15 \
  Arete-27B-BF16.gguf Arete-27B-AD-Q4_K_M.gguf Q8_0

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

Fine-tuned from Qwen/Qwen3.8-27B by Nous Research (NousResearch/Arete-27B, revision 108f0afb), Apache-2.0.

64 layers (16 full attention, 48 linear attention), hidden size 5120, feed-forward size 17408, vocabulary 248,320, context 262,144. No multi token prediction head.

Needs a recent llama.cpp with qwen35 architecture support. Built and tested with llama.cpp b12205 (a691b2345).