Ornith-1.5-122B for coding on a 24 GB GPU (expert-pruned GGUF)

A coding-focused build: experts were chosen on calibration data that was mostly code (70% of the mix is code instructions, coding-agent trajectories and function calls, and all of Ornith's own reasoning replies answer coding prompts), and only coding was benchmarked.

Run Ornith-1.5-122B fully on a 24 GB card. unsloth's smallest Ornith GGUF (UD-IQ1_M) is 37.6 GB, more than a 24 GB GPU holds before any context. This GGUF removes half of the routed experts in every layer, so the whole model and a 32K context fit on a 24 GB GPU. It is cut from unsloth's 2-bit UD-IQ2_XXS GGUF, and nothing is re-quantized: every tensor that remains is byte for byte unsloth's.

Sizes are in GB the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card. The Hugging Face file list counts in decimal, so it shows this file as 21.42 GB.

This build keeps 99.4% of the reference model's HumanEval+ score and 99.3% of its MBPP+ score (95% CIs: 96.1 to 102.6% on HumanEval+, 95.7 to 103.0% on MBPP+) at 26.9% of its file size. The reference is unsloth's 4-bit UD-Q4_K_XL (the 8-bit Q8_0 is too large for the machine these were evaluated on), so these figures combine the cost of pruning with the cost of going from 4-bit to 2-bit; the unpruned 2-bit model was not evaluated.

Files

REAP50 means 50% of each layer's routed experts were kept.

Run it

Use a llama.cpp build with Qwen3.5-MoE (qwen35moe) support. Evaluated with llama.cpp at commit 36cbf415, plus a small local patch that returns malformed chat output as plain text instead of an HTTP error; it never triggered in any run behind this card.

llama-server -m Ornith-1.5-122B-24GB-REAP50-UD-IQ2_XXS.gguf -ngl 99 -c 32768 \
  -ctk q8_0 -ctv q8_0 -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0

• Ornith reasons before answering by default. The sampling settings are ornith-ai's recommended ones for coding, and the ones the benchmarks used. Its chat template can turn reasoning off (--chat-template-kwargs '{"enable_thinking": false}'); that mode was not evaluated.
• Set a token limit on replies: the pruned model sometimes keeps reasoning without finishing (see the table). On HumanEval+ its median reply was 999 tokens against 1,010 for the reference, and its mean 1,832 against 1,306.
• Memory was measured with these settings: one slot, every layer on the GPU, 8-bit KV cache, flash attention on. With a 16-bit KV cache (drop -ctk, -ctv) it allocates 20.8 GB at 32K context.
• The multi-token-prediction (MTP) block is removed, so this file cannot be used for MTP speculative decoding. No vision projector is included, and vision was not tested.
• LM Studio and Ollama have not been tested.

How it was made

Ornith-1.5-122B routes each token to 8 of 256 experts in each of its 48 MoE layers, plus one shared expert; the routed experts are 93% of its parameters. We measured how much each expert contributes on 3.54 million tokens: a 2,047-conversation calibration mix (code instructions 35%, general chat 30%, software-engineering agent trajectories 17.5%, function calling 17.5%; 3.08M tokens) and 128 of Ornith's own reasoning replies to coding prompts from that mix (0.46M tokens), all run through unsloth's UD-Q4_K_XL. Experts were scored with REAP (router weight times expert output size, averaged over the tokens routed to the expert), and the top 128 of 256 were kept in every layer. Every layer keeps the same number because llama.cpp's loader reads one expert count for the whole model. The shared expert, both attention types (full attention and Gated DeltaNet), the embeddings and the output head are untouched.

The pruning is a byte copy: kept experts, their router rows, and every other tensor are identical to unsloth's file, checked tensor by tensor (879 tensors). Expert scores were stable: rankings from the two halves of the calibration set correlate at a median of 0.988 across layers (0.967 in the least stable layer).

Method background: Half the Experts, All the Code, accepted at the COLM 2026 Workshop on Efficient Reasoning.

Evaluation details

• Code: EvalPlus HumanEval+ (164 problems) and MBPP+ (378), served by llama-server, reasoning on, temperature 0.6, top-p 0.95, top-k 20, min-p 0, one fixed seed per problem, one sample per problem, up to 32,768 generated tokens. The answer after the reasoning block was sanitized and scored with EvalPlus 0.3.1. A reply that hit the token limit counts as a failure.
• Against the reference: −0.6 points on HumanEval+ (95% CI −3.7 to +2.4; 4 problems lost, 3 gained, McNemar p = 1.000) and −0.5 points on MBPP+ (95% CI −3.4 to +2.4; 18 problems lost, 16 gained, McNemar p = 0.864). Neither difference is statistically significant (McNemar p >= 0.05 for both), which does not show the two models are equal. Paired bootstrap CIs and per-problem results are in eval/.
• The EvalPlus sanitizer timed out on 1 answer (HumanEval/69); a timed-out answer is scored as a failure.
• One sample per problem: the CIs cover the choice of problems, not run-to-run sampling noise.
• Memory: GPU memory of the llama-server process (nvidia-smi), model fully offloaded, one slot, read once the server was ready and before any request, measured on an NVIDIA DGX Spark (unified memory). llama.cpp's own breakdown at 32K context with the 8-bit KV cache: 20,004 MB weights + 557 MB KV cache and recurrent state + 238 MB compute. Memory on an actual 24 GB discrete GPU was not measured; on a GPU that also drives a display, leave some headroom.

Limitations

• Only single-function coding (HumanEval+, MBPP+) was measured; agentic coding, tool calls, long-context use, vision, general knowledge and other languages were not, and may degrade more.
• The pruned model hits the token limit more often than the reference (see the table).

License and attribution

Unofficial community derivative of Ornith-1.5-122B, pruned from unsloth's GGUF quantizations; not affiliated with or endorsed by ornith-ai or Unsloth. Distributed under the MIT License, the license both source repositories declare. Neither repository includes a license file, so LICENSE holds the standard MIT text from SPDX. The GGUF metadata (general.name "Ornith 1.5 122B", general.license "mit") is carried over unchanged from unsloth's files.