Ornith-1.5-122B IQ3_XXS-B GGUF (for a 16 GB GPU + 64 GB RAM)

A 51.7 GB GGUF of ornith-ai/Ornith-1.5-122B (MIT), sized so the whole model fits a 16 GB GPU plus a 64 GB RAM PC with nothing read from the SSD while it writes. The routed experts are IQ3_XXS with an importance matrix; everything else (attention, the delta-net projections, the shared experts) is kept at Q8_0, since that is where most of the loss of a stock IQ3_XXS came from.

日本語の説明は下にあります。

File

Two shards (llama-gguf-split), since the Hub takes files up to 50 GB: point llama.cpp or Abyss at the first one. The NextN (MTP) block (blk.48, in the second shard) is kept at Q4_K, for engines that draft with it. token_embd is IQ3_S (llama-quantize's pick for an IQ3_XXS target): Abyss v1.3.56 and earlier silently left the NextN drafter off on this file because of it (found by a user report); use Abyss v1.3.57 or later. llama.cpp is not affected.

Quality (KL against BF16)

llama-perplexity --kl-divergence against the BF16 conversion, 8 chunks of 1024 tokens of a held-out English and Japanese text (not the imatrix text). BF16's own perplexity on it: 14.44.

This is a lossy 3-bit file: it trades about 3.5x the KL of UD-Q4_K_XL for 28 GB less, which is what lets it stay off the SSD on a 64 GB PC. If your GPU + RAM holds 80 GB, UD-Q4_K_XL is the better file.

Speed

Abyss v1.3.57 ./abyss.sh --bench --deep, on one PC set up as a 16 GB GPU + 64 GB RAM machine without an iGPU: an AMD Radeon AI PRO R9700 capped at 16 GB (STRATA_GLM_VRAM_GB=16), a 46 GB RAM tier inside a 64 GB memory limit, the experts the GPU misses computed on the CPU (Ryzen AI Max+ 395, 16 threads), PCIe Gen4 x4, the NextN drafter on. Greedy, 3 answers of 256 tokens.

The whole file fits the GPU + RAM, so nothing is read from the SSD while it writes; UD-Q4_K_XL leaves ~21 GB of experts on the SSD on such a PC. The CPU here has fast memory (LPDDR5X); a desktop with dual-channel DDR5 computes the missed experts more slowly, so expect less there. And a 16 GB card that also drives the screen has ~14.5 GB for the model, not 16. A user's RX 9070 XT 16 GB + 64 GB dual-channel DDR5 desktop measured 31-34 tok/s on v1.3.55 (the NextN drafter off, see above).

Before v1.3.57 (the NextN drafter off on this file) the same PC measured 39.5 tok/s, 39% of the n-gram drafts accepted.

Coding tasks (llmbench-style, one run each)

Each task is a small repository with a bug report or a feature request and hidden tests; "resolved" means the tests pass. Same Abyss server, temperature 0.6 / top_p 0.95 / top_k 20.

Both missed t021, t044 and t046. One run each, so a task or two either way is noise: read it as "no visible loss on coding tasks", not as better than UD-Q4_K_XL. The speed in these runs (v1.3.55, the NextN drafter off): 33-38 tok/s on most tasks, 22-25 on the longest multi-file answers.

How it was made

llama.cpp bed0a8566 (build 11370):

python convert_hf_to_gguf.py <download of ornith-ai/Ornith-1.5-122B> --outtype bf16 --outfile Ornith-1.5-122B-BF16.gguf
llama-imatrix -m Ornith-1.5-122B-BF16.gguf -f calib.txt -c 2048 -b 8192 -ub 8192 --chunks 32 -o imatrix-ornith122b.gguf
llama-quantize --imatrix imatrix-ornith122b.gguf \
  --tensor-type 'blk\.48\.=q4_K' \
  --tensor-type 'attn_=q8_0' --tensor-type 'ssm_=q8_0' --tensor-type 'shexp=q8_0' \
  Ornith-1.5-122B-BF16.gguf Ornith-1.5-122B-own-IQ3_XXS-B.gguf IQ3_XXS 16
llama-gguf-split --split --split-max-size 30G Ornith-1.5-122B-own-IQ3_XXS-B.gguf Ornith-1.5-122B-IQ3_XXS-B

• blk.48 (the NextN block) is never run while the imatrix is computed, so it has no importance data and has to be kept above the very low-bit types (Q4_K here).
• The imatrix text (0.7 MB) is English and Japanese technical writing and code. It is not uploaded.

Running it

• llama.cpp (qwen35moe): llama-server -m Ornith-1.5-122B-IQ3_XXS-B-00001-of-00002.gguf -ngl 99 --n-cpu-moe <n>, choosing <n> so the dense weights and as many experts as fit sit on the GPU.
• Abyss: pack it with tools/iq_pack.py --gguf <dir>/Ornith-1.5-122B-IQ3_XXS-B-00001-of-00002.gguf --out <dir>/pack (the pack indexes the shards in place) and use the config of docs/ornith/README.md; the engine reads the NextN drafter from the shards. On a 16 GB GPU + 64 GB RAM PC, STRATA_GLM_RAM_GB ~46.

Sampling as the model card: temperature 0.6, top_p 0.95, top_k 20.

The same recipe on Qwen3.5-122B-A10B: Qwen3.5-122B-A10B-IQ3_XXS-16G64G-GGUF (KL 0.035, 41 tok/s on the same setup).

License

MIT, as the original model (model card). This repository only holds a quantized conversion; all credit for the model goes to its authors.

---

日本語

ornith-ai/Ornith-1.5-122B（MIT）を 51.7 GB の GGUF にしたものです。 GPU 16 GB + RAM 64 GB の PC に全体が収まり、生成中に SSD を読まないサイズを狙いました。 ルーティングされるエキスパートは imatrix 付きの IQ3_XXS で、それ以外（attention、delta-net の射影、共有エキスパート）は Q8_0 のまま残しています。標準の IQ3_XXS で劣化の大部分が出ていたのが、ここだったためです。

• 品質：BF16 との KL は 0.138（UD-Q4_K_XL は 0.039）。3 bit なので劣化はあります。 GPU + RAM に 80 GB 載る PC なら、UD-Q4_K_XL のほうが良いです。
• 速度（Abyss v1.3.57、GPU 16 GB・RAM 64 GB・iGPU なしを想定した設定）：生成 42.6 tok/s、NextN の下書きの採用率 71% （UD-Q4_K_XL は 20.9 tok/s、SSD 待ちが 37%）。 テストに使った CPU はメモリが速い（LPDDR5X）ので、DDR5 デュアルチャネルのデスクトップではこれより遅くなります。 画面もつないだ 16 GB のカードでは、モデルに使えるのは 14.5 GB ほどです。
• Abyss v1.3.56 以前では、このファイルで NextN の下書きが動いていませんでした（token_embd が IQ3_S のため。利用者の報告で判明）。 v1.3.57 以降を使ってください。そのときの速度は 39.5 tok/s（n-gram の下書き）でした。llama.cpp には影響ありません。
• コーディング課題（各 1 回）：60 問中 56 問、難しい 16 問（L7）で 12 問を解決（UD-Q4_K_XL は 60 問中 54 問、L7 は 11 問）。 1 回ずつの計測なので「目に見える劣化はない」と読んでください。
• 2 つのファイルに分けています（Hub は 1 ファイル 50 GB まで）。1 つ目のファイルを指定してください。
• NextN（MTP）の層（blk.48、2 つ目のファイル）は Q4_K で入っています。
• 作り方・速度・ライセンスは上の英語の節のとおりです。
• 同じ作り方の Qwen3.5-122B-A10B 版：Qwen3.5-122B-A10B-IQ3_XXS-16G64G-GGUF（KL 0.035、同じ設定で 41 tok/s）