Ornith-1.5-122B for coding on a 16 GB GPU (expert-pruned GGUF + recovery adapter)
Run Ornith-1.5-122B fully on a 16 GB card. unsloth's smallest Ornith GGUF (UD-IQ1_M) is 37.6 GB. This repo is a pruned GGUF plus a small adapter that is loaded with it. Together they need 13.6 GB of GPU memory with a 32K context.
A model pruned this far often gets stuck repeating itself while it reasons. The recipe here has three parts, and every score on this card for this build was measured with all three:
- The adapter (
--lora-scaled), trained to bring the pruned layers closer to the unpruned model. - Two sampling flags (
--presence-penalty 1.5 --repeat-last-n 20480). - Stop and redraw (
redraw.py, included): a reply that starts repeating is stopped and drawn again with a new seed, up to three draws.
| Ornith UD-Q4_K_XL (reference) | This repo | |
|---|---|---|
| File | 74.2 GB | 11.9 GB + 1.2 GB adapter |
| GPU memory at 32K context, 8-bit KV cache (measured at server start) | 71.9 GB | 13.6 GB |
| GPU memory at 32K context, 16-bit KV cache (measured at server start) | 72.2 GB | 13.9 GB |
| GPU memory at 128K context, 8-bit KV cache (measured at server start) | 73.4 GB | 15.1 GB |
| HumanEval+ (fraction solved, reasoning on) | 0.945, one draw | 0.884 with redraw, 0.860 on the first draw |
| MBPP+ (fraction solved, reasoning on) | 0.810, one draw | 0.712 with redraw, 0.693 on the first draw |
Sizes are in GB the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card. The Hugging Face file list counts in decimal, so it shows the files as 12.76 GB and 1.25 GB.
The two score columns were not measured the same way. The reference had one draw per problem and no penalty flags; this build used the recipe above. On the first draw, counting a reply that was stopped for repeating or hit the token limit as a failure, this build solved 141 of 164 HumanEval+ problems and 262 of 378 MBPP+ problems, against 155 and 306 for the reference (91.0% and 85.6% of its counts). With redraw it solved 145 and 269 (93.5% and 87.9%). The reference is unsloth's 4-bit UD-Q4_K_XL, so the gap includes both the pruning and the step from 4-bit to 2-bit.
What the adapter adds (LiveCodeBench)
Measured on 130 LiveCodeBench problems (easy and medium, releases v1 to v3) that were set aside before any model here generated on them, and used only for this one test. Both rows use the same flags and the same stop-and-redraw client.
| Solved of 130 | On the first draw | |
|---|---|---|
| The plain 25% prune (64 experts per layer, no adapter) | 104 | 89 |
| This repo (66 experts per layer + adapter) | 116 | 100 |
- Gain: +9.2 points (95% CI +3.1 to +15.4; 15 problems gained, 3 lost; McNemar p = 0.0075). On the first draw alone: +8.5 points (95% CI +0.8 to +16.2).
- By difficulty: easy 62 of 63, medium 54 of 67 (plain prune: 60 of 63 and 44 of 67).
- Not compared with the unpruned model: it was not run on these problems, and hard problems were not run at all.
Costs to know about
- MBPP+ is slightly lower with the adapter: 269 of 378 against 276 for the plain prune with the same flags and redraw (−1.9 points, 95% CI −5.6 to +1.6; 20 gained, 27 lost; McNemar p = 0.382). The difference is not statistically significant, which does not show the two are equal. On HumanEval+ it is 145 against 144 of 164.
- It was slower than the plain prune in our test: the adapter is not merged into the quantized weights; llama.cpp computes its correction while the model runs. The LiveCodeBench test took 1.7 times the wall time of the plain prune (16,655 s against 9,815 s at 32 parallel slots) for 1.12 times the tokens. Single-user speed was not measured.
- It still loops sometimes: 22 of 130 first draws were stopped for repeating (31 for the plain prune). After three draws, 4 problems still ended in a loop and 2 hit the token limit.
Files
| File | Use |
|---|---|
Ornith-1.5-122B-16GB-REAP25-UD-IQ2_XXS.gguf |
The pruned model, 66 experts per layer. Pruned from unsloth's UD-IQ2_XXS (39.1 GB) |
recovery-adapter/Ornith-1.5-122B-16GB-REAP25-adapter.gguf |
Required. Low-rank corrections to the 64 kept experts in every layer |
redraw.py |
Stop-and-redraw client for llama-server. Python 3.9+, standard library only |
selections/ornith-reap25-plus2.json |
The experts kept and restored in every layer |
REAP25 means 25% of each layer's routed experts were kept by the REAP score (64 of 256);
two more per layer were restored and retrained, so the file holds 66.
Run it
Use a llama.cpp build with Qwen3.5-MoE (qwen35moe) support. Evaluated with llama.cpp at commit
36cbf415, plus a small local patch that returns malformed chat output as plain text
instead of an HTTP error; it triggered 0 times in the runs behind this card.
llama-server -m Ornith-1.5-122B-16GB-REAP25-UD-IQ2_XXS.gguf \
--lora-scaled recovery-adapter/Ornith-1.5-122B-16GB-REAP25-adapter.gguf:1 \
-ngl 99 -c 32768 -ctk q8_0 -ctv q8_0 -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
--presence-penalty 1.5 --repeat-last-n 20480
python3 redraw.py --url http://127.0.0.1:8080 "Write a Python function that ..."
- Load the adapter. The two restored experts in each layer were trained together with it, and the GGUF on its own was not evaluated.
- Keep the two penalty flags. They are part of every score on this card.
- Use
redraw.py, or the same rule in your own client. It streams the reply and stops it when the text ends in the same block six times in a row (blocks of up to 6,000 characters; a short block must repeat for at least 200 characters). A reply that was stopped, or that hit the token limit, is asked again with a new seed, for at most three draws. A reply that finishes is never redrawn. Chat apps do not do this for you; without it, expect the first-draw numbers, and looping replies that run to the token limit. - Ornith reasons before answering by default. The sampling settings are ornith-ai's recommended
ones for coding. Its chat template can turn reasoning off
(
--chat-template-kwargs '{"enable_thinking": false}'); that mode was not evaluated. - Set a token limit on replies (32,768 tokens per draw in every run here).
- Memory was measured with these settings: one slot, every layer on the GPU, 8-bit KV cache, flash
attention on. With a 16-bit KV cache (drop
-ctk,-ctv) it allocates 13.9 GB at 32K context. At 128K context it allocates 15.1 GB, which leaves little room on a 16 GB card. - The multi-token-prediction (MTP) block is removed, so this file cannot be used for MTP speculative decoding. No vision projector is included, and vision was not tested.
- LM Studio, Ollama and other llama.cpp versions have not been tested.
How it was made
Ornith-1.5-122B routes each token to 8 of 256 experts in each of its 48 MoE layers, plus one shared expert. Experts were scored with REAP (router weight times expert output size, averaged over the tokens routed to the expert) on 3.54 million tokens of calibration data weighted toward coding, and the top 64 of 256 were kept in every layer. The scores and the data are those of the 24 GB build, whose card describes the mix; every expert here is also in that build.
- Kept experts are untouched. The 64 kept experts, their router rows and every other tensor are byte for byte unsloth's UD-IQ2_XXS; nothing kept is re-quantized. Checked in two steps: the plain prune against unsloth's file (879 tensors), then this file against the plain prune (687 tensors whole, and the kept part of the 192 expert and router tensors).
- Two experts per layer are restored and retrained. The next two experts in each layer's ranking were put back with their original router rows. Their weights were trained in full precision and quantized again with llama.cpp's quantizer, each tensor in the format unsloth's file uses for it.
- The adapter corrects the kept experts. Each kept expert gets a rank-8 correction to its three weight matrices. It lives in the adapter file and llama.cpp adds it while the model runs, so the quantized bytes stay as they are.
- Training. One layer at a time, the corrections and the restored experts were fitted so that the layer's expert output matches what the unpruned model's experts (unsloth's 4-bit UD-Q4_K_XL) produce on the same input. The data were calibration documents and the plain prune's own long reasoning replies to 124 other LiveCodeBench problems, none of them among the 130 test problems above. No coding score was used during fitting.
The shared expert, both attention types (full attention and Gated DeltaNet), the embeddings and the output head are untouched.
Method paper: Reasoning Again with a Quarter of the Experts: Retry First, Then Train.
Pruning background: Half the Experts, All the Code, accepted at the COLM 2026 Workshop on Efficient Reasoning.
Evaluation details
- LiveCodeBench: official prompt, answer extraction and grader, served by llama-server with 32 parallel slots, reasoning on, temperature 0.6, top-p 0.95, top-k 20, min-p 0, one fixed seed per problem and draw, up to 32,768 generated tokens per draw, at most three draws. The comparison, its decision rule and the problem list were fixed in a hashed file before the test ran.
- A second, smaller check before the test: on 64 other set-aside problems, each run twice with different seeds, the gain was +6.2 points (95% CI +0.8 to +13.3).
- HumanEval+ (164 problems) and MBPP+ (378): same server settings and redraw rule, answers sanitized and scored with EvalPlus 0.3.1. On the first draw the plain prune solved 138 and 258, this build 141 and 262.
- Completeness: every run of this build and of the plain prune behind this card finished every problem with 0 failed requests.
- One sample per problem per arm: the CIs cover the choice of problems, not run-to-run sampling noise.
- Memory: GPU memory of the llama-server process (nvidia-smi), model fully offloaded, one slot, read once the server was ready and before any request, measured on an NVIDIA DGX Spark (unified memory). llama.cpp's own breakdown at 32K context with the 8-bit KV cache, printed before the adapter is loaded: 11,750 MB weights + 557 MB KV cache and recurrent state + 238 MB compute. Memory on an actual 16 GB discrete GPU was not measured; on a GPU that also drives a display, leave some headroom.
Limitations
- Only coding was measured (LiveCodeBench easy and medium, HumanEval+, MBPP+). Agentic coding, tool calls, long-context use, vision, general knowledge and other languages were not, and may degrade more.
- The three-draw numbers need a client that stops and redraws.
- The costs listed above: a small MBPP+ loss against the plain prune, slower generation in our test, and loops that still occur.
License and attribution
Unofficial community derivative of Ornith-1.5-122B, pruned
from unsloth's GGUF quantizations; not affiliated with or
endorsed by ornith-ai or Unsloth. Distributed under the MIT License, the license both source
repositories declare. Neither repository includes a license file, so LICENSE holds the standard
MIT text from SPDX. The GGUF metadata (general.name "Ornith 1.5 122B", general.license "mit") is
carried over unchanged from unsloth's files.