Ornith-1.5-122B-MLX-Mixed2-3-8bit-MTP

A mixed-precision MLX build of Ornith-1.5-122B, converted on Apple Silicon (M3 Max, 64 GB) and runnable with stock mlx_lm or served through oMLX. These are quantization variants, not fine-tunes — no training of any kind was performed on these weights.

Routed experts carry the compression: ffn_gate_exps and ffn_up_exps at 2-bit, ffn_down_exps at 3-bit (all affine, group size 64). Attention, the GatedDeltaNet blocks, the shared expert, embeddings and head stay at 8-bit. Norms, conv1d, A_log and dt_bias remain BF16.

The MTP (multi-token prediction) head is included as a grafted 4-bit shard, so the model can run native speculative decoding.

Model Details

Quantization Details

Produced by a streaming GGUF reader that dequantizes one layer at a time and re-quantizes with mx.quantize (affine, group size 64), so peak RAM stays near one layer and no 244 GB BF16 intermediate is ever materialized.

Allocation mirrors AtomicChat's published AD-* ladder rules.

Measured requantization error (dequantized MLX vs. the Q8_0 source, relative L2):

Dequantization is lossless; requantization costs the full price of the target width on top of the source's own error, so a port can never beat its source. The Q8_0 source sits at KL 0.0096 vs BF16 (96.28% top-1) — roughly two orders of magnitude below the 2-bit expert error, so it is not the limiting factor here.

MTP Head (grafted, 4-bit)

Measured draft quality — agreement between the MTP's argmax and the main model's actual next-next token, over 19 positions with full context:

63% top-1 is in line with AtomicChat's reported 59–62% draft acceptance for the same head on the 35B-A3B sibling.

MTP depth — measured, and depth 2 is optimal

The MTP is a single layer chained autoregressively for depth > 1. Measured through oMLX on M3 Max 64 GB, 300-token generations, temperature 0, one model reload per setting, short context (~15 tokens):

Acceptance decays gently rather than collapsing, so depth 2 pays for itself and depth 3 does not. MTP costs ~120 ms against ~6100 ms of backbone (2%) — the win is entirely in the extra confirmed tokens per verify, not in cheap drafting.

Per-depth figures are conditional: d2 is measured only where d1 was accepted, as chained MTP requires.

The gain shrinks with context length — use the 16k number if you serve long prompts

Long-context figures are the mean of 4 runs per arm (spread ≤1%, stats byte-identical across runs).

Acceptance is context-independent — the head drafts just as well at 14k. The gain shrinks because the cost moves: at short context decode is weight-read-bound, so folding 2–3 verify tokens into one weight read is nearly free. With a long KV cache, attention dominates and scales with the number of tokens verified, so the verify consumes most of what batching saved.

Plan for ~1.1× on long prompts and ~1.4× on short ones. MTP costs 1.42 GB and is well worth it in both cases, but do not expect the short-context figure to hold on document or code workloads.

Independently, the MTP head's top-1 agreement with the main model's real next-next token measured 63% (top-5: 95%) over 19 positions at full context — consistent with the ~64% live acceptance above.

Enabling MTP requires the key in text_config

For oMLX, mtp_num_hidden_layers must be inside text_config, not only at the top level:

{ "text_config": { "mtp_num_hidden_layers": 1, "mtp_use_dedicated_embeddings": false } }

oMLX's patch reads it via TextModelArgs.from_dict(text_config); with it only at the top level the count is 0, the head is never attached, and MTP silently does nothing — the model loads, the log reports Lightning MTP (..., active), and the draft path never engages. Every diagnostic on that path is logged at DEBUG, so at the default log_level: info there is no error at all.

To confirm MTP is actually drafting (not merely selected), look for:

MTP path activated for uid=0 (model has mtp_forward, batch=1, primed=5)
MTP[0] finish=length tokens=300 cycles=132 tok/cycle=2.27 accept=166/270 (61.5%)
       depth[d1=86/132,d2=52/86,d3=28/52]

Absence of MTP path activated means the head is attached but idle.

Do not confuse MTP with expert offload

A large apparent MTP speedup (6.5 → 32 tok/s) is easy to misattribute. In the case observed here it was MoE expert offload to SSD, not MTP:

moe expert offload: wrapped 48 layers at 25.0% residency
MoE offload: hit_rate=33.3% misses=45747 fetched=152901.8 MB  (prompt 15933, output 45)

152.9 GB fetched from disk to produce 45 tokens. That is the entire slowdown.

Worse, on qwen3_5_moe the two settings are mutually exclusive — oMLX's MOE_OFFLOAD_MTP_MODEL_TYPES covers only deepseek_v41, glm5_next and qwen4_exp, so enabling expert offload forces Lightning MTP off (and raises if both are requested). They therefore look causally linked when they are only mechanically linked. If you are benchmarking MTP, turn expert offload off first, and check fetched= in the scheduler log to confirm it is not streaming from SSD.

Note: enabling MTP requires the MTP keys to be listed in model.safetensors.index.json. Stock mlx_lm ignores them (sanitize() strips mtp.*), but oMLX's detection reads the index and will silently skip the head if they are absent.

Validation

Conversion was verified tensor-by-tensor against the original HF weights:

• Structural: 2,153 emitted tensors, 0 missing, 0 extra, 0 shape mismatches
• Numeric: A_log and dt_bias match HF to 0.000000; conv1d matches to 100% exact
• Generation: correct output on factual, arithmetic, code, reasoning, JSON-only and German prompts

Getting the GatedDeltaNet parameters right required three separate transforms, since llama.cpp stores them differently from HF/MLX: a value-head grid transpose ([ratio, key_head] vs [key_head, ratio]), a log(-x) on ssm_a, and the same permutation applied to the conv1d channels. Feeding the GGUF through untransformed loads cleanly and passes every shape check while generating looping garbage.

Usage

pip install mlx-lm

python3 -m mlx_lm generate --model Ornith-1.5-122B-MLX-Mixed2-3-8bit-MTP \
  --prompt "Explain GatedDeltaNet in two sentences." --max-tokens 200 --temp 0.6

Recommended sampling: temperature 0.6, top-k 20, top-p 0.95. Thinking is enabled by default; pass chat_template_kwargs: {"enable_thinking": false} to disable it per request.

Peak memory is ~47.8 GB for the text model. With the MTP head loaded, budget ~49.2 GB — on a 64 GB machine this is close to the ceiling, and any concurrent memory use will cause paging and a large throughput drop.

Lineage

ornith-ai/Ornith-1.5-122B            125.1 B params, BF16
  └─ AtomicChat/Ornith-1.5-122B-GGUF  Q8_0, 132.6 GB  (KL 0.0096 vs BF16)
       └─ this repo                   mixed 2/3-bit affine, 49.02 GB  (~2.9 bpw)

Base Model Capabilities

Ornith-1.5-122B is a 125B-parameter sparse MoE with a 3:1 GatedDeltaNet / full-attention hybrid backbone. It supports a 262,144-token context, thinking mode with a thinking block, tool calling, multilingual input, and a native multi-token prediction head. The base checkpoint is also image-capable; vision is not included here (see below).

Attribution

Converted, verified and documented by Hermes Agent.

The quantization allocation follows AtomicChat's published AD-* ladder rules, and their importance-matrix calibration corpus informed the layout.

License

MIT, inherited from ornith-ai/Ornith-1.5-122B.

Related Models

Limitations

• Text only. Vision weights are excluded.
• Peak memory ~49.2 GB. Tight on a 64 GB machine.
• oMLX needed for MTP. Stock mlx_lm strips the MTP head and runs without speculative decoding.
• MTP's gain depends on context length — ~1.4× on short prompts, ~1.1× at 14k. See above.
• No KL-divergence or perplexity figure is published for this build; the requantization errors above are per-tensor measurements, not an end-to-end quality evaluation.
• The MTP head is 4-bit; the source GGUF stores it at Q8_0.
• Throughput figures come from one M3 Max (64 GB); other hardware will differ.