Run Ornith-1.5-122B fully on a 16 GB card. unsloth's smallest Ornith GGUF (UD-IQ1_M) is 37.6 GB. This repo is a pruned GGUF plus a small adapter that is loaded with it. Together they need 13.6 GB of GPU memory with a 32K context.
A model pruned this far often gets stuck repeating itself while it reasons. The recipe here has three parts, and every score on this card for this build was measured with all three:
--lora-scaled), trained to bring the pruned layers closer to the unpruned model.--presence-penalty 1.5 --repeat-last-n 20480).redraw.py, included): a reply that starts repeating is stopped and drawn
again with a new seed, up to three draws.| Ornith UD-Q4_K_XL (reference) | This repo | |
|---|---|---|
| File | 74.2 GB | 11.9 GB + 1.2 GB adapter |
| GPU memory at 32K context, 8-bit KV cache (measured at server start) | 71.9 GB | 13.6 GB |
| GPU memory at 32K context, 16-bit KV cache (measured at server start) | 72.2 GB | 13.9 GB |
| GPU memory at 128K context, 8-bit KV cache (measured at server start) | 73.4 GB | 15.1 GB |
| HumanEval+ (fraction solved, reasoning on) | 0.945, one draw | 0.884 with redraw, 0.860 on the first draw |
| MBPP+ (fraction solved, reasoning on) | 0.810, one draw | 0.712 with redraw, 0.693 on the first draw |
Sizes are in GB the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card. The Hugging Face file list counts in decimal, so it shows the files as 12.76 GB and 1.25 GB.
The two score columns were not measured the same way. The reference had one draw per problem and no penalty flags; this build used the recipe above. On the first draw, counting a reply that was stopped for repeating or hit the token limit as a failure, this build solved 141 of 164 HumanEval+ problems and 262 of 378 MBPP+ problems, against 155 and 306 for the reference (91.0% and 85.6% of its counts). With redraw it solved 145 and 269 (93.5% and 87.9%). The reference is unsloth's 4-bit UD-Q4_K_XL, so the gap includes both the pruning and the step from 4-bit to 2-bit.
Measured on 130 LiveCodeBench problems (easy and medium, releases v1 to v3) that were set aside before any model here generated on them, and used only for this one test. Both rows use the same flags and the same stop-and-redraw client.
| Solved of 130 | On the first draw | |
|---|---|---|
| The plain 25% prune (64 experts per layer, no adapter) | 104 | 89 |
| This repo (66 experts per layer + adapter) | 116 | 100 |
| File | Use |
|---|---|
Ornith-1.5-122B-16GB-REAP25-UD-IQ2_XXS.gguf |
The pruned model, 66 experts per layer. Pruned from unsloth's UD-IQ2_XXS (39.1 GB) |
recovery-adapter/Ornith-1.5-122B-16GB-REAP25-adapter.gguf |
Required. Low-rank corrections to the 64 kept experts in every layer |
redraw.py |
Stop-and-redraw client for llama-server. Python 3.9+, standard library only |
selections/ornith-reap25-plus2.json |
The experts kept and restored in every layer |
REAP25 means 25% of each layer's routed experts were kept by the REAP score (64 of 256);
two more per layer were restored and retrained, so the file holds 66.
Use a llama.cpp build with Qwen3.5-MoE (qwen35moe) support. Evaluated with llama.cpp at commit
36cbf415, plus a small local patch that returns malformed chat output as plain text
instead of an HTTP error; it triggered 0 times in the runs behind this card.
llama-server -m Ornith-1.5-122B-16GB-REAP25-UD-IQ2_XXS.gguf \
--lora-scaled recovery-adapter/Ornith-1.5-122B-16GB-REAP25-adapter.gguf:1 \
-ngl 99 -c 32768 -ctk q8_0 -ctv q8_0 -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
--presence-penalty 1.5 --repeat-last-n 20480
python3 redraw.py --url http://127.0.0.1:8080 "Write a Python function that ..."
redraw.py, or the same rule in your own client. It streams the reply and stops it when
the text ends in the same block six times in a row (blocks of up to 6,000 characters; a short
block must repeat for at least 200 characters). A reply that was stopped, or that hit the token
limit, is asked again with a new seed, for at most three draws. A reply that finishes is never
redrawn. Chat apps do not do this for you; without it, expect the first-draw numbers, and looping
replies that run to the token limit.--chat-template-kwargs '{"enable_thinking": false}'); that mode was not evaluated.-ctk, -ctv) it allocates 13.9 GB at 32K context.
At 128K context it allocates 15.1 GB, which leaves little room on a 16 GB card.Ornith-1.5-122B routes each token to 8 of 256 experts in each of its 48 MoE layers, plus one shared expert. Experts were scored with REAP (router weight times expert output size, averaged over the tokens routed to the expert) on 3.54 million tokens of calibration data weighted toward coding, and the top 64 of 256 were kept in every layer. The scores and the data are those of the 24 GB build, whose card describes the mix; every expert here is also in that build.
The shared expert, both attention types (full attention and Gated DeltaNet), the embeddings and the output head are untouched.
Method paper: Reasoning Again with a Quarter of the Experts: Retry First, Then Train.
Pruning background: Half the Experts, All the Code, accepted at the COLM 2026 Workshop on Efficient Reasoning.
Unofficial community derivative of Ornith-1.5-122B, pruned
from unsloth's GGUF quantizations; not affiliated with or
endorsed by ornith-ai or Unsloth. Distributed under the MIT License, the license both source
repositories declare. Neither repository includes a license file, so LICENSE holds the standard
MIT text from SPDX. The GGUF metadata (general.name "Ornith 1.5 122B", general.license "mit") is
carried over unchanged from unsloth's files.
ornith-ai/Ornith-1.5-122B