Ornith-1.5-122B in EXL3, 4.0 bits per weight

These are EXL3 weights of Ornith AI's Ornith-1.5-122B at 4.0 bits per weight, 64.7 GB in all. We converted them from the BF16 checkpoint, ornith-ai/Ornith-1.5-122B at revision e09bc0dfbe, with exllamav3 1.6.2 and a small fix to its quantizer (in convert/). Every projection is at 4 bits: the routed experts, the shared expert, the Gated DeltaNet layers and the attention layers alike. The output head is at 6 bits and the MTP layer at 4. The vision tower is in the files too, quantized to 6 bits. Only the embedding stays at 16 bits.

We serve these weights with TandemLLM, our inference engine for one DGX Spark. Ornith-1.5-122B is a Qwen3.5 mixture-of-experts model (Qwen3_5MoeForConditionalGeneration) with 48 layers: 36 Gated DeltaNet and 12 gated attention. Each layer has 256 routed experts, of which 8 run a token, plus one shared expert. One more layer predicts the next tokens (MTP). The vocabulary has 248,320 tokens and the context is 262,144 tokens. From the tensor shapes, one decoded token reads about 4.76 GB of these weights.

Quality against BF16

We checked the pack against the BF16 checkpoint twice, on plain wikitext text. Neither run uses the chat template or a thinking block. KL is the mean KL divergence from BF16 a position, and top-1 is how often both pick the same next token.

The model_diff run is in quality/model_diff.txt. Its median KL a token is 0.032 and its 90th percentile 0.224, so a few hard positions carry most of the mean. On the 8 rows TandemLLM reads the pack with its own kernels and lands within 0.003 KL of exllamav3's own forward.

Dead experts

13 routed experts in layers 3 to 13 have all-zero weights once cast to fp16: their BF16 values sit near 1e-35. Stock exllamav3 1.6.2 stops on them, because refitting the scales of an all-zero weight solves a singular system. The patch in convert/ keeps the input scales in that case and changes nothing else. These experts convert to zero output scales, so a routed token gets exactly 0 from them. On the 8 rows above, TandemLLM's router sent 0.22 % of all routed picks to them.

Speed on a DGX Spark

One DGX Spark (GB10, 128 GB), nothing else running. TandemLLM at commit 88b8070e, the code release 0.13.0 is built on. One request at a time, 512 greedy tokens, thinking off, a fresh server for each row. The game file row asks for a whole HTML game and runs 2,048 tokens. Edit and copy are asks whose answer repeats much of the prompt. Each cell is one run, in tok/s.

Speed is greedy with thinking off. A fresh TandemLLM install thinks at effort medium; docs/setup.md says how to change it and which effort each published number used. In every drafted run the text was the same as plain greedy, token for token. Both drafted rows add TandemLLM's lookup, which drafts from text already in the prompt. Both use anchor backoff: after a round where StairCut drafted nothing, the engine takes 1, 2, 4, then 8 plain steps before it drafts again. StairCut sets the block size from the pack's price file, the measured cost of a verify step by block size. The DFlash2 row uses incoai/Ornith-1.5-122B-DFlash2 with its layers in NVFP4 and a 32k hot vocabulary.

A start loads the pack in about 65 s. The weights take 59.95 GiB in memory, 61.16 GiB with the MTP layer. After the load the Spark still had about 47 GB of MemAvailable. Prompts read at 1,217 to 1,442 tok/s at 3,664 and 4,597 tokens, one request at a time.

Tool calls

We ran tool-eval-bench at commit f2e368b against this pack on TandemLLM 0.13.0, on one DGX Spark on 2026-10-21. The server had one slot and its default effort at xhigh, and every row used --seed 42 --temperature 0. A fresh install runs at medium. The raw rows send their own effort, and the chat profile sets each turn's own, so the tools rows are the ones that depend on the server's level: set REASONING_EFFORT=xhigh to match them (docs/setup.md). Each row is one run, and no answer came from a cache. The full suite has 69 scenarios, and hard mode has 28.

The chart also shows the two test profiles from TandemLLM's docs, which do not ship.

TC-68 failed in every row: it asks for a value the prompt leaves out, and the model does not invent one. Every row, its misses and the flags are in TandemLLM's docs/tool-eval-bench.md.

How to run it

TandemLLM runs this pack from release 0.13.0 on. On a DGX Spark with TandemLLM 0.13.0 installed:

~/TandemLLM/run/ops/use-profile.sh ornith-1.5-122b-exl3-4.0

use-profile.sh downloads this repository at a pinned commit, checks each file against SHA256SUMS and starts the engine on port 8000 with the MTP layer as the draft head. QSE_ORN_HEAD=dflash2 switches to the DFlash2 drafter, which the profile fetches.

These are standard EXL3 files. The patch changes only the quantizer, and the Qwen3.5 code that reads the files is upstream's, so stock exllamav3 1.6.2 loads them. We have not tested other EXL3 loaders.

How we made it

convert/exllamav3-1.6.2-ornith.patch applies to exllamav3 v1.6.2 (commit 4c84273). convert/README.md has the commands. We used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook, on two RTX PRO 6000 GPUs; the conversion took 81 minutes. quality/convert.txt is the log.

License

Ornith-1.5-122B's model card gives its license as MIT. Its repository holds no LICENSE file, so LICENSE here is the standard MIT text, and NOTICE says where the license comes from, where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed too. Everything else about the model, including its intended use and its benchmarks, is in Ornith AI's model card.

Credits

Ornith AI made Ornith-1.5-122B and its BF16 checkpoint. incoai made the DFlash2 drafter for it (Apache-2.0). turboderp made exllamav3 and the EXL3 format. TandemLLM is ours.