Ornith-1.5-397B — IQ3_XXS GGUF

An IQ3_XXS quantization of ornith-ai/Ornith-1.5-397B.

The official Ornith-1.5-397B-GGUF repository stops at Q4_K_M (224.08 GiB). That does not fit in 192 GB of VRAM. This one does.

Split into 4 files of ≤45 GB. Point llama.cpp at -00001-of-00004 and it loads all four.

---

Files

Total 1098 tensors, 153,194,399,488 bytes across 4 files (142.67 GiB; the unsplit file is 153,194,399,008 B — the difference is per-split headers). See SHA256SUMS.txt.

For vision, use mmproj-Ornith-1.5-397B-BF16.gguf from the official GGUF repo (not mirrored here).

---

How it was made

Source was the official Q8_0 GGUF (392.56 GiB, 8.51 BPW), not the BF16 checkpoint. That means --allow-requantize was used — this is a re-quantization of an already-quantized tensor set. Q8_0 is close to lossless, but this is stated plainly so you can weigh it.

llama-quantize --allow-requantize \
  --imatrix <imatrix.gguf> \
  --token-embedding-type q5_K \
  Ornith-1.5-397B-Q8_0.gguf Ornith-1.5-397B-IQ3_XXS.gguf IQ3_XXS 64

llama-gguf-split --split --split-max-size 45G \
  Ornith-1.5-397B-IQ3_XXS.gguf Ornith-1.5-397B-IQ3_XXS

--token-embedding-type q5_K overrides the IQ3_XXS default (iq3_s) for token_embd. With a 248,320-token vocabulary carrying CJK, the extra ~250 MiB is worth it.

Quantization took 31m40s on a Threadripper PRO 9985WX (64 cores, 64 threads).

About the importance matrix

The imatrix is not ours and is not mirrored here. We used unsloth/Qwen3.5-397B-A17B-GGUF's imatrix_unsloth.gguf_file (80 chunks × 11264 tokens).

This works because Ornith-1.5-397B is a light fine-tune of Qwen/Qwen3.5-397B-A17B:

• the 1371 non-MTP tensor names are identical sets (set difference is empty)
• the vision tower is bit-identical (frozen), as are linear_attn.A_log and dt_bias
• the language trunk has cosine similarity 0.9993–0.99999 (relative L2 of 1–4%)
• the safetensors total_size differs by exactly 13,191,153,536 B — precisely the MTP head

We verified name compatibility before quantizing: 765 of 765 imatrix entries match tensors in the Ornith Q8_0 (100%). The 180 quantizable tensors without imatrix coverage are norms and ssm_conv1d, which are not quantized anyway.

Notably, 765 is the same quantize.imatrix.entries_count recorded in the official Ornith GGUF headers — the official build used the same number of entries.

If you want a purpose-built imatrix, compute one against this model directly. We did not, and we say so rather than implying otherwise.

---

Measured

Pure CPU, Threadripper PRO 9985WX, 64 threads, -dev none:

Same prompt (three summer haiku, different kigo, one line each), temp 0.8, thinking off:

Q8_0 —

金魚売り通り過ぎていく水の音
青トマトかじれば夏の朝の味
夕立やアスファルト跳ねる子らの声

IQ3_XXS —

夏日や池の鯉ゆく水草かげ
夏炉や炉の灰に眠る火の粉かな
夏空や雲の切れ間より富士の山

Both hold 5-7-5 and use three distinct summer kigo. Q8_0 reaches for more modern imagery, IQ3_XXS sits closer to classical form. Neither is broken.

Perplexity has not been measured. Stated as missing rather than guessed at.

Why not IQ2

We also baked IQ2_XXS (97.65 GiB, 2.12 BPW) and do not recommend it. It answers factual questions correctly ("日本の首都は東京です") but cannot carry out multi-step generation — asked for haiku it emits bullet-point glossaries of season words, and at temp 0.8 it degenerates into repetition with stray tokens. At 2.12 BPW this model does not survive. It is not published here.

IQ3_XXS is, in our measurements, the floor.

---

Usage

llama-server -m Ornith-1.5-397B-IQ3_XXS-00001-of-00004.gguf \
  -c 32768 --threads 64

Ornith is a reasoning model and it thinks at length. With -n 1500 it had not finished deliberating. For direct answers:

--chat-template-kwargs '{"enable_thinking":false}'

If you keep thinking on, budget generously (the 35B sibling needed ≥6500 tokens) and strip everything before </think> before parsing code out of a response — otherwise you will grade the model's scratch work instead of its answer.

⚠️ GPU offload does not work yet on SM 12.0

On 12× RTX PRO 2000 Blackwell (SM 12.0, CUDA 13.2) this model crashes on GPU:

ggml_cuda_compute_forward: SOFT_MAX failed
CUDA error: invalid argument

Isolated by bisecting -ngl:

• -ngl 1 (layer 59, a full_attention layer) → runs
• -ngl 2 (adds layer 58, a linear_attention layer) → crashes

So it is the linear-attention (gated delta net) path. -fa on does not help (flash_attn = enabled is logged and SOFT_MAX is still reached), nor does --no-warmup, nor -ub 1 -b 1. Reproduced on both a 2026-08-10 build and on master at d59d455 (174 commits newer). CPU inference is unaffected.

Separately, llama.cpp misclassifies Blackwell as an integrated GPU because cudaDeviceProp.integrated is non-zero (the driver API correctly reports 0 for the same device). Only the first "iGPU" is kept, so -sm/-ts silently do nothing and everything piles onto device 0. Upstream #26901, open since 2026-08-11. Work around it by naming devices explicitly:

-dev CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7,CUDA8,CUDA9,CUDA10,CUDA11
-ts 4.5,5,5,5,5,5,5,5,5,5,5,6.5

That does distribute the layers correctly (verified in the load log) — the SOFT_MAX crash is a separate, unresolved problem.

Note for anyone re-converting from safetensors

config.json declares mtp_num_hidden_layers=1, but there is not a single MTP tensor in the checkpoint (1371 tensors, 0 MTP) or in the official GGUF (1098 tensors, 0 nextn). The 35B-A3B sibling does ship 785 of them; the 397B does not, in either 1.0 or 1.5.

Convert with --no-mtp. Without it you get a GGUF declaring block_count=61 with an empty blk.60, and llama.cpp fails at load with a missing-tensor error.

---

Attribution

• Base model: ornith-ai/Ornith-1.5-397B — MIT. All credit for the model belongs to its authors.
• Importance matrix: unsloth/Qwen3.5-397B-A17B-GGUF.
• Tooling: llama.cpp.

This repository contributes quantized weights and the measurements above. Nothing else.