Ornith-1.5-122B - EXL3 - EAQ v1 - 4.0 bpw

Quantization of ornith-ai/Ornith-1.5-122B (source revision 38c106ad) to about 4 bits per weight with EAQ v1.

Format: standard EXL3. EAQ v1 is the standard EXL3 format (quant_method: exl3, codebook mul1; tensors stored as trellis / suh / svh / mul1) produced by the EAQ conversion. It loads in exllamav3; the tree used for conversion and for every measurement on this page is based on exllamav3 v1.6.1 (other exllamav3 versions were not tested).

Summary

Bit layout (MEASURED)

Structure check against the source checkpoint (MEASURED): all 1,949 source tensors are accounted for, 0 missing and 0 unexpected quantized tensors, 0 per-tensor width mismatches, and all 76,658 floating-point tensors in the pack are finite.

Quality versus BF16 (MEASURED)

Evaluation. 100 held-out conversations (code, tool calling, agentic, reasoning, chat, multilingual), 92,067 scored tokens, compared with the BF16 model. For every token position the full next-token distribution of the BF16 model (teacher) is compared with the quantized model's (student): KL(BF16 || quant) in nats, whether the arg-max token agrees (top-1 agreement), and the negative log-likelihood of the actual next token under each model. Scored tokens are the assistant turn (reasoning text, answer text, tool calls, end of turn); prompt context is not scored. Both models ran with TF32 disabled, over contexts of up to 4,096 tokens. Note: the evaluation prompts were rendered with the Qwen3.8 chat template, not Ornith's own template, so absolute numbers can differ for prompts in Ornith's native format. Confidence intervals are 95% bootstrap intervals that resample whole conversations.

Headline

The quantized model's NLL on the reference text is within 0.003 nats of the BF16 model's (slightly lower, which is within noise and not a claim of improvement). No other quantization of this model was measured, so there is no comparison to another quant.

By category

Categories prefixed "thinking:" are conversations that contain a reasoning trace.

Per-category intervals are wide because each category has only 4 to 15 conversations; differences between neighbouring categories are mostly within noise. Tool calling and chat are the highest-KL categories and code the lowest.

Grouped view

By token type

Tool-call structure, tool names and argument keys agree with BF16 on 99.5%, 98.8% and 99.2% of tokens; argument values (free-form strings and numbers) are the least stable tool-call part, with the second-highest KL of any token type.

KL distribution

The mean is dominated by a small tail: the median token has KL 0.0011, while 0.061% of tokens have KL above 1.

Error through the depth of the network

Relative hidden-state error against BF16 after each decoder layer (each model running on its own states) grows smoothly with depth, from 0.5% after layer 0 to about 14% at the output of layer 47, peaking at about 23% around layers 43-44. Expert routing agreement follows the same trend: the router's top-1 expert matches BF16 for 99.1% of tokens in layer 0 and 78.4% in layer 47 (mean over layers 90.3%).

Sanity test (MEASURED)

In exllamav3: the model loads in 30 s using 63.4 GiB of VRAM, and greedy generation is correct on 4 of 4 test prompts (capital of France, 17 x 23, a Python add function, the first five primes) and is deterministic across a repeat run. These are smoke tests, not benchmarks.

Note on the source checkpoint: 13 all-zero experts

13 routed experts in the source checkpoint have weights so small (about 1e-18 to 1e-34) that they are exactly zero in fp16. They are kept as exact zeros in this quant (verified after conversion: dequantized values are exactly 0.0 and finite). The affected experts, as (layer, expert index):

Usage

Requires exllamav3 (tested with a v1.6.1-based tree). The model needs about 64 GiB of VRAM plus cache.

from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/Ornith-1.5-122B-EXL3-EAQv1-4-bpw")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=32768)
model.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

messages = [{"role": "user", "content": "Write a Python function that reverses a linked list."}]
prompt = tokenizer.hf_render_chat_template(messages, add_generation_prompt=True)
print(generator.generate(prompt=prompt, max_new_tokens=512, completion_only=True, encode_special_tokens=True))

Recommended sampling (from the base model's generation_config.json): temperature 0.6, top-k 20, top-p 0.95. Stop tokens: 248046 and 248044.

Files

model-0000N-of-00009.safetensors (weights), model.safetensors.index.json, config.json (with quantization_config), quantization_config.json (per-tensor storage map), tokenizer and template files, generation_config.json, preprocessor configs, and images/ (the charts on this page). Tokenizer, chat template, generation config and preprocessor files are byte-identical to the base model's.

Not tested

Vision inputs, MTP speculative decoding, long-context behaviour, throughput, and public benchmark scores have not been measured for this quant.

Credits

• Ornith AI for the base model, Ornith-1.5-122B.
• turboderp for exllamav3 and the EXL3 format.
• Quantized by MiaAI Lab with the EAQ conversion.