The checkpoint · Speed · Quality · What it fits on · Use it · The base model · Bench card

Ornith-1.5-122B-A10B-W4A16-Merlin

A 4-bit build of Ornith 1.5 122B, ornith-ai's fine-tune of Qwen's Qwen3.5-122B-A10B (a 122B mixture-of-experts model, 10B active per token, with vision), made to run at its full 262,144-token context on four 24 GB GPUs. It is 66 GiB instead of the original 233 GiB: the routed experts are INT4, the Gated DeltaNet projections INT8, and attention, the shared experts and the vision tower stay in BF16. It is served by the Merlin engine, a vLLM 0.32.0 fork tuned for Ampere and Ada, with the model's own MTP head for speculative decoding and the whole KV cache in VRAM. Measured on RTX 3090s, with and without P2P; the same profiles also match RTX 3090 Ti, RTX A5000 and other four-card sm_86 24 GB setups (untested), and no other GPUs have profiles for this checkpoint yet (see What it fits on).

_{numbers: Merlin 3.0.1 on 4× RTX 3090 w/o NVLINK @ 220W, P2P on and the GPU unlock, 2026-11-01 to 2026-11-02, on release candidates just before the final 3.0.1 commit. KV pool: FP8, all in VRAM; three ~262K needle prompts sent at once, each needle found (3.0.0 run, 2026-10-21). Agents: eight 16K conversations, then eight simultaneous 512-token follow-ups, thinking off. Decode: one stream, thinking on in full (Ornith's template has no effort levels), exactly 261,888 prompt tokens + 128 out, median of 3. Prefill: one cold 101,845-token prompt. Each figure is also in a table below.}

The checkpoint

Files in this repository

_{About 66 GiB in all (130 files, listed with their SHA-256 in SHA256SUMS; this card and its images are not listed, since card edits change them). ornith-ai's card declares MIT and their repository ships no licence text, so LICENSE-MIT carries the standard MIT text; the Qwen base model's Apache-2.0 licence is in LICENSE-QWEN.}

Speed

_{4× RTX 3090, one stream, median of 3. Prompt tokens: thinking on 4,046 / 32,566 / 130,386, thinking off 4,105 / 32,625 / 130,445, and exactly 261,888 at 262K for both; the time-to-first-token column is the thinking-on sends. Thinking on: ≤ 256 tokens out (128 at 262K), in full: the template has no effort levels. Thinking off: the answer task, natural stop, ≤ 1,024 tokens out (128 at 262K). Below 262K every send was a cold prefill; at 262K send 1 was cold and sends 2 and 3 reused the prefix before the needle; the 262K time to first token is send 1's. A separate 259,557-token ladder row decodes at 99 tok/s (thinking on).}

_{One cold request per prompt, one stream, thinking on, 8 tokens out; the 261,888-token row is the 2026-11-01 run's cold thinking-on send at 262K (128 tokens out), so it can differ slightly from the 262K row above.}

_{N 16K conversations, then N simultaneous 512-token follow-ups, total tok/s while all N decode. The 8-agent rows were re-measured on Merlin 3.0.1 (temperature 0, one fresh boot per layout); the 2- and 4-agent rows are from the released Merlin 3.0.0 install, 2026-10-21, and were not re-run. With thinking on, a 512-token follow-up may end inside the reasoning and reasoning tokens count, so that total measures decoded tokens, not finished answers. Each overlap window was short (8.1 s thinking off and 7.4 s thinking on; 5.6 and 7.0 s for the 2- and 4-agent rows), so read these rows loosely.}

_{Three needle prompts sent at once (Merlin 3.0.0 run, 2026-10-21; not re-run for 3.0.1, and not a residency test): the engine ran all three (running peak 3, KV pool use 100 %, 0 errors) and found every needle; first tokens came at 155, 305 and 454 s as each prefill finished, so no combined decode rate is given.}

Against the previous release (3.0.0)

_{tok/s. Decode: one stream, median of 3, same frozen driver and prompts; thinking on is the template's full thinking in both releases. Prefill: one cold send each. Cold time to first token at 262K: 115.8 s against 153.7 s. Not a like-for-like comparison: the 3.0.0 column was measured on the Merlin 3.0.0 release install with Ornith's engine changes added before they merged (that build also carried a prefill experiment that did not ship), and its agents figure on the released 3.0.0 install, all with P2P and without the GPU unlock; the 3.0.1 column on Merlin 3.0.1 with P2P and the unlock. Ornith had no stock-versus-unlocked comparison, so how much of the prefill gain comes from the unlock is not measured. Thinking-off decode at 131K is the one cell that is lower.}

How it was measured

• Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02. The 3.0.1 runs (2026-11-01 to 2026-11-02) also had the GPU unlock installed (Zagitta's gpu-unlock, an optional host upgrade); the 3.0.0 runs (2026-10-21) had P2P only.
• Engine: Merlin 3.0.1, served by merlin serve with the shipped defaults, one fresh boot per layout; the engine wheel is the same file as in 3.0.0. The rows were measured on release candidates just before the final 3.0.1 commit, not on the final commit. Rows marked (3.0.0 run) were measured on the Merlin 3.0.0 release install with Ornith's engine changes added before they merged into 3.0.0 (that build also carried a prefill experiment that did not ship).
• Layout: TP2 × PP2 + EP on four cards, Gated DeltaNet RecoverSSM, MTP with 3 draft tokens from the model's own head, uniform MoE verify on, FP8 KV, 1,024-token prefill chunks. Thinking in full: the template's only switch is enable_thinking, so Merlin's default effort medium has no effect on this checkpoint.
• Decode: T=0, a needle prompt at depth; one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
• MTP acceptance (tokens per step, K=3, served sampling, thinking on; 3.0.0 run): prose 2.39-2.48, code 2.56, GSM8K 2.58.
• Streams are client-timed. Recipe, settings and limits of the evidence: the engine's Ornith 1.5 bench card; the rows marked (3.0.0 run) are on the 3.0.0 bench card.

Quality

This checkpoint, measured on the Merlin engine:

_{GSM8K thinking on: Merlin 3.0.1 on four cards, 2026-11-01 to 2026-11-02, the same 200 problems, 5-shot, same scorer, temperature 0; none was cut at the 8,192-token cap (mean 261 tokens, longest 3,634). Rows marked (3.0.0 run) were measured on 2026-10-21 on the Merlin 3.0.0 install with the Ornith engine changes before they merged (GSM8K on a separate boot of the same setup, and 190 / 200 again on the released 3.0.0 install) and were not re-run for 3.0.1; the weights are unchanged. The two GSM8K rows differ in thinking mode, so they are not a 3.0.0 to 3.0.1 comparison. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made, so how much quality the 4-bit build keeps is not quantified here.}

The fine-tune, as reported by ornith-ai for the BF16 weights:

_{A selection from ornith-ai's card, which has the full table, the harness settings and comparisons with other models. Not re-run by us or on this 4-bit build.}

What it fits on

_{Measured on the reference host (Merlin 3.0.0 run, 2026-10-21; not re-run for 3.0.1, and the KV pool and the sessions it holds are unchanged): three ~262K needle prompts (259,261–261,565 tokens) sent at once, each needle found (whether the pool kept all three resident was not tested); an 8 MP (8,388,608 px) image with a 262K prompt left at least 1,058 MiB free on every card (checked with uniform MoE verify off; the speed figures run with it on).}

Other GPUs. The engine also ships profiles for these; the Status column says how far each has been run:

_{merlin serve and ./setup.sh pick the profile from the GPUs they find. A ~ is the engine's estimate, not a measurement. Both profiles use four cards and an FP8 KV cache; the no-P2P row was measured on a rented host (4× RTX 3090, PCIe x8, 2026-10-21, Merlin 3.0.0 run) with its own KV pin (977,838-token pool). There are no profiles yet for RTX 4090s, 32 GB and 48 GB cards, A100s, A30s or CMP 170HX. 🟢 measured by us · ⚫ a profile ships, not yet run by us (expected to work). Every profile: the engine's hardware matrix.}

Use it

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model ornith1.5

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8000. Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again. Requests use the model name ornith.

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

Manual install

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \
  bash release/install-env.sh ./venv

venv/bin/hf download halt95/Ornith-1.5-122B-A10B-W4A16-Merlin --revision 7116c9e4c469286850f46230ce266edab8206c3f \
  --local-dir ~/models/ornith1.5
(cd ~/models/ornith1.5 && sha256sum -c SHA256SUMS)
venv/bin/python merlin_cli.py serve ~/models/ornith1.5

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8000. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels.

Recommended settings (ornith-ai's, for this model):

_{ornith-ai's recommendation, with min_p=0.0 and repetition_penalty=1.0. Merlin's defaults are the checkpoint's generation_config.json (temperature 0.6, top_p 0.95, top_k 20) with presence and repetition penalties of 0 and 1.0. Reasoning tokens and the answer share the max_tokens budget.}

• Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
• No effort levels: the template reads only enable_thinking, so reasoning_effort (the Qwen3.8 checkpoints' switch, where Merlin's default is medium) has no effect on this checkpoint, and thinking runs in full.
• Chat template: Merlin serves ornith-ai's template as shipped, with the qwen3 reasoning parser and the qwen3_coder tool-call parser. It rejects a system message that is not the first message: clients that send per-turn system notes get an error, and clients that edit the system prompt in place lose prefix-cache reuse.
• Up to 8 images per prompt, up to 8,388,608 px each. Video is off in Merlin's launcher.
• --kv-ram-cache SIZE, the engine's experimental host-RAM KV cache (new in 3.0.1, off by default), is accepted for this checkpoint but untested ⚫.

The base model

Ornith 1.5 is ornith-ai's fine-tune of Qwen3.5-122B-A10B. The architecture, in short:

• Hybrid attention. Three Gated DeltaNet (linear-attention) layers for every full-attention layer: 36 and 12 of the 48 layers.
• Sparse experts. 256 experts per layer with 8 routed and 1 shared active per token: 10B of the 122B parameters do the work for each token.
• Multi-token prediction. One MTP layer, its own head (it differs from Qwen's); Merlin uses it for speculative decoding.
• Vision. A 27-layer vision transformer, unchanged from Qwen's in our checks: the fine-tune trained the language model and the MTP head.
• Thinking by default, with a non-thinking mode per request.

_{Architecture from the checkpoint's config.json; the parameter count is the Hub's for ornith-ai's repository. Which weights the fine-tune changed: our reads of sampled tensors against Qwen3.5-122B-A10B.}

Other builds

_{One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.1, 2026-11-02 bench cards, measured on release candidates just before the final commit with P2P and the GPU unlock. Reasoning effort medium, Merlin's default, on every Qwen3.8 checkpoint; the Nemotron and Ornith 1.5 templates have no effort levels, so they ran at full thinking.}

Good to know

• The 3.0.1 figures come from one setup, 4× RTX 3090 at 220 W, PCIe Gen4 with P2P and the GPU unlock; the rows marked (3.0.0 run) were measured on the same host with P2P only. The one other host is the no-P2P row under Other GPUs, measured on a rented host; the Other builds table names each build's card count. Hosts without P2P use the no-P2P profile (see What it fits on).
• Four cards are the minimum.
• Independent derivative: not made, reviewed or endorsed by ornith-ai or Qwen. ornith-ai's model card covers the model's training, intended use and evaluation; read it before use.

Changes

_{Built on the reference host; the Hub repository starts with this build. Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.}

Credits and licence

ornith-ai (the model; MIT, as their card declares, see LICENSE-MIT), Qwen (the base model, Qwen3.5-122B-A10B; Apache-2.0, see LICENSE-QWEN), the vLLM project (the engine Merlin is built on), aikitoria (peer-to-peer on consumer Ampere), Neiko2002, and CoolShades (the idle-power report behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.