Ornith-1.5-122B · GSQ-RCO IQ3_S GGUF

A personal quantization, made on end-user hardware. One person built this file on one home desktop (AMD Ryzen 9 5900X, 64 GB of RAM, an RTX 3090 and an RTX 3070 Ti, a 4 TB NVMe SSD), not on a lab cluster. It is not an official release of IST Austria, the Ornith team or the Qwen team: it applies their published work (see Credits).

A mixed-precision GGUF of ornith-ai/Ornith-1.5-122B (a 122B mixture-of-experts model on the Qwen3.5-122B-A10B architecture, ~10B active parameters per token), quantized with GSQ and RCO, the two methods of IST Austria's DAS Lab, following the procedure of their GSQ-RCO GGUF releases:

• GSQ (Gumbel-Softmax Quantization) refines the integer codes of every tensor inside a GGUF type, against the inputs that tensor actually sees, at every candidate type;
• RCO (Riemannian Constrained Optimization) then picks one type per tensor, under an exact size budget, by minimizing the KL divergence of the whole model's output from the BF16 model's.

The budget is the one ISTA uses for their IQ3_S releases: 3.50 bits per weight over the whole file. The result is not a uniform IQ3_S: each tensor carries the type RCO assigned to it, from IQ2_XS to Q6_K.

Files

This repository holds the language model only, split in two parts under Hugging Face's file-size limit. Load the first part; llama.cpp finds the second one next to it.

Ornith-1.5-122B also has a vision encoder and a multi-token-prediction (MTP) head, which this file does not contain:

• Vision. Image input needs a vision projector (mmproj) GGUF loaded next to the model. The projector is not part of this quantization, so any projector of Ornith-1.5-122B works, for example bartowski's mmproj-Ornith-1.5-122B-bf16.gguf, or one converted from the original checkpoint with llama.cpp's convert_hf_to_gguf.py --mmproj.
• MTP. The MTP head drafts tokens for speculative decoding. llama.cpp's convert_hf_to_gguf.py --mtp exports it from the original checkpoint as a separate draft GGUF (Q8_0 suits it), which llama-server takes with -md.

Quality

Mean KL divergence and same-top-1 rate of the next-token distribution against the BF16 GGUF of the same model, 100 chunks of 512 tokens per domain, on three domains held out from the GSQ and RCO calibration mix:

What RCO chose

Dense tensors mostly stay high: attention, DeltaNet projections and the shared expert are Q6_K for the most part (a few Q5_K, Q4_K, IQ4_XS, IQ3_S), token_embd and output are Q6_K, ssm_alpha/ssm_beta are kept in BF16, and norms, router and convolution weights in F32. The routed experts, which are most of the file, get the low types, and RCO spends the budget unevenly across depth:

Tensor counts over the file: Q6_K 265, IQ3_S 61, IQ4_XS 35, IQ2_XS 25, Q5_K 19, IQ3_XXS 17, Q4_K 13, IQ2_S 11, BF16 72, F32 361. The type of every tensor can be read from the file itself, for example with gguf-py's gguf-dump.

How it was made

1. Conversion. The Hugging Face checkpoint was converted to a BF16 GGUF with llama.cpp's convert_hf_to_gguf.py, without the MTP head.

2. Candidates. Every tensor was quantized at each candidate type (IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q4_K, Q5_K, Q6_K) with llama.cpp's llama-quantize and bartowski's importance matrix of this model (573 chunks of 512 tokens, every expert of every layer covered).

3. Hessians. The BF16 model was run on 192 sequences of 2,048 tokens of the calibration mix (below), one decoder layer at a time, collecting the input Hessian H = XᵀX / n of every linear layer, and a separate Hessian per routed expert from the tokens routed to it.

4. GSQ. Each candidate tensor was refined inside its GGUF type: the integer codes (for the K-quant types, a shift of −2…+2 per weight; for the IQ codebook types, the grid entries nearest each 4- or 8-value group) are chosen by a Gumbel-softmax relaxation that minimizes the layer's output error ‖(W̃ − W)·L‖²_F with H = L·Lᵀ, while the block scales and signs stay as llama.cpp set them (200 steps, Lion, lr 6e-4). A refined tensor is kept only if its hard-rounded error went down. Experts are refined each against its own Hessian, shrunk toward the layer's pooled Hessian in proportion to how few tokens reached it; an expert routed fewer than 64 calibration tokens keeps llama.cpp's codes. Mean reduction of the layer error against llama.cpp's own imatrix quantization:

   IQ2_XS	IQ2_S	IQ3_XXS	IQ3_S	IQ4_XS	Q4_K	Q5_K	Q6_K
   30.2 %	31.1 %	34.2 %	41.6 %	40.0 %	34.7 %	36.3 %	40.3 %

5. RCO. One type per tensor group (350 groups; the gate and up projections of a layer's experts share one) was searched under the 3.50 bpw whole-file budget, minimizing KL(BF16 ‖ quantized) over the full vocabulary on 256 calibration sequences of 2,048 tokens, with RCO's own optimizer code (Gumbel straight-through samples under the budget, Riemannian Adam on the budget manifold, its sensitivity initialization and final knapsack rounding; 200 steps, 4 samples per step, temperature 1.0 → 0.01, lr 0.1). The model, too large for memory, was streamed one decoder layer at a time, and the gradient of the choice was formed from each layer's inputs and output gradients over the GSQ candidates.

6. Assembly. The chosen GSQ candidates were copied into one file as they are.

ISTA publishes the methods and their code, but neither GGUF tooling for GSQ nor an allocation for this architecture: here GSQ was reimplemented for GGUF types after the paper, and RCO was run through its own code.

Hardware and time

Everything ran on the desktop named above. The BF16 weights (about 245 GB) are almost four times its RAM, so the Hessian collection, GSQ and RCO all stream the model one decoder layer at a time, and the candidate files (the eight types, then their GSQ refinements, about 1 TB together) live on the SSD. GSQ shared its work between the two GPUs and took about 41 hours; RCO ran on the RTX 3090 alone, about 11.5 hours for its 200 steps; conversion, candidates and Hessians took a few hours more.

Calibration mix

Documents of five domains interleaved, so that any prefix of the text is balanced, in the spirit of GSQ's calibration mixture: WikiText-2 train, Python source code, OpenThoughts-114k (problems, reasoning traces and solutions), Open-Platypus (instruction/response pairs, the first 2,000 rows held out for evaluation), and the text of the GSQ and RCO papers.

Usage

Any llama.cpp build recent enough for qwen35moe loads it. For example, with the expert weights kept in system RAM and everything else on the GPU:

llama-server -m Ornith-1.5-122B-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
  --mmproj mmproj-Ornith-1.5-122B-bf16.gguf \
  -c 262144 -ngl 99 --cpu-moe --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0

--mmproj is only needed for image input, and -md with an MTP draft GGUF adds speculative decoding (see Files); this command, with both, was checked on upstream llama.cpp (build 11669, commit 620d2225c). --n-cpu-moe N keeps only the first N layers' experts in RAM, for GPUs with room for more. The sampling above is Ornith's recommendation for precise coding; for general tasks it recommends temperature 1.0 with presence_penalty 1.5. Ornith is a reasoning model: its answers open with a <think> block.

Credits

• Ornith team for Ornith-1.5-122B, the model quantized here.
• Qwen team for Qwen3.5 and the architecture this model builds on.
• IST Austria, DAS Lab for GSQ and RCO (GSQ code, RCO code) and for the GSQ-RCO GGUF procedure and size budget this file follows.
• llama.cpp / ggml for the GGUF format, its quantization types, conversion and quantization tools, and the runtime used for evaluation.
• bartowski for the importance matrix of this model and its vision projector GGUF.
• The authors of WikiText-2, OpenThoughts-114k and Open-Platypus, used for calibration and evaluation.

Citation

@misc{ornith_1_5,
    title = {{Ornith-1.5}: From Self-Scaffolding to Self-Improvement},
    url = {https://ornith.ai/ornith_1_5.html},
    author = {{Ornith Team}},
    year = {2026}
}
@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

MIT, as the base model. Qwen3.5, which it builds on, is under Apache 2.0.