Ornith-1.0-35B-NVFP4-MTP-graft

What this is: the agentic-coding fine-tune deepreinforce-ai/Ornith-1.0-35B (a Qwen3.6-35B-A3B MoE, 256 experts / 8 active) made fast on NVIDIA GB10, plus a fix for a chat-template defect that causes repeating tool-call loops in agentic CLIs.

Three changes over upstream:

1. MTP head graft. Ornith-1.0-35B was published without the MTP weights (the config declares MTP, but the safetensors index has no mtp.* tensors). We graft the 19 mtp.* tensors from nvidia/Qwen3.6-35B-A3B-NVFP4 (same architecture) onto Ornith. This enables multi-token prediction (self-speculative decoding, K=3) with ~100% acceptance — the draft and verify share the model, so the fine-tune's behaviour is preserved.
2. lm_head W4. The BF16 lm_head (1.0 GB over 248,320 vocab, read every decode step) is re-quantized to 4-bit (compressed-tensors WNA16 int4 g32, served by Marlin).
3. Chat-template fix (see below). We ship the canonical Qwen3.6 template instead of upstream Ornith's, which re-injects stale reasoning into history.

Vision: the tower is intact (this card previously failed to say so)

Qwen3.6-35B-A3B is a vision-language model, and so is this checkpoint. Until 2026-08-13 this card declared pipeline_tag: text-generation and never mentioned the vision tower — that was wrong, and it is what this section fixes.

Quantization deliberately skips the whole tower (recipe.yaml: ignore: ['re:.*visual.*', …]), so it survives in BF16, unquantized:

So nothing in this repo blocks the vision path. We have not exercised it ourselves — every number below is text-only, on a text-only agentic-coding workload. If vLLM logs

Model … is treated as multimodal but has no registered multimodal processor; running in text-only mode.

that warning comes from vLLM's architecture resolution, not from the weights: it means the model class vLLM resolved has no _processor_factory attached. Qwen3_5MoeForConditionalGeneration has been in the multimodal section of vLLM's model_executor/models/registry.py since v0.25.0, so a stock ≥0.26 wheel should resolve it. Check the Resolved architecture: line in your startup log — if it is anything other than Qwen3_5MoeForConditionalGeneration (in particular if you also see "has no vLLM implementation, falling back to Transformers implementation"), the fix is on the engine side, not the checkpoint.

If you are bringing vision up, drop --speculative-config first. MTP self-speculative decoding together with multimodal inputs is a combination we have never measured. Confirm an image round-trips, then put MTP back.

⚠️ The chat-template defect we fixed (relevant to anyone using Ornith-1.0-35B)

Upstream deepreinforce-ai/Ornith-1.0-35B ships a chat_template.jinja that removed the conditional guarding <think> re-rendering for past assistant turns, while leaving the ns.last_query_index computation in place (an unfinished edit). Verified by hash: upstream Ornith's template is md5 fa5eb1ee…, whereas Qwen/Qwen3.6-35B-A3B and nvidia/Qwen3.6-35B-A3B-NVFP4 are byte-identical to each other (md5 52b6d51a…). The only difference is:

{#- canonical Qwen3.6 (correct): strip thinking from older turns #}
{%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
    {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
{%- else %}
    {{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}

{#- upstream Ornith: ALWAYS re-renders old <think> #}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}

Effect: every prior assistant turn's chain-of-thought is replayed into the prompt, so the model re-reads a stale plan ("I should call tool X") and re-issues it — reported as constant tool-call loops with agentic clients (e.g. qwen-code). It also bloats context. This repo ships the canonical template; if you use upstream Ornith directly, pass vLLM --chat-template with the Qwen3.6 template.

Also worth knowing: Qwen forbids greedy decoding. temperature 0 does not make tool-calling safer — it causes degradation and infinite repetition loops. For quantized checkpoints use temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.

Result (measured on NVIDIA GB10 / DGX Spark, sm_121, single stream, 512 output tokens, text-only)

Prefill @131K: 47.8 s (vs 518 s pre-optimization, 10.8×). TTFT cached @131K: 1.9 s. MTP acceptance ~100% at K=3. K=4 is a regression (accept rate at position 4 falls below breakeven). Tooling quality preserved: 346/370 on our internal 370-case battery (BF16 head scored 343/370 — within noise). Upstream Ornith agentic-coding scores (per its card): SWE-bench Verified 75.6, SWE-bench Pro 50.4, Terminal-Bench 2.1 64.2, Claw-eval 69.8.

Serving (vLLM on GB10, custom sm121 build)

vllm serve pocharlies/Ornith-1.0-35B-NVFP4-MTP-graft \
  --served-model-name ornith-1.0-35b-nvfp4-mtp \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching --enable-chunked-prefill \
  --enable-auto-tool-choice --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --attention-backend flashinfer \
  --max-num-seqs 4 --max-num-batched-tokens 32768 --gpu-memory-utilization 0.55 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --trust-remote-code

env: VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_NVFP4_GEMM_BACKEND=marlin

This command is what we run for text-only agentic coding, and it is what every number on this card was measured with. It does not disable vision — but see Vision above: if you want images, bring the engine up without --speculative-config first.

KV cache nvfp4 is impossible on GB10. FlashInfer requires sm100f (Blackwell datacenter); GB10 is sm121 → the engine aborts. Use --kv-cache-dtype fp8.

Attribution & license

MIT (inherited from Ornith-1.0-35B). Ornith weights © deepreinforce-ai (MIT); the 19 grafted mtp.* tensors are from nvidia/Qwen3.6-35B-A3B-NVFP4 (Apache-2.0); base architecture Qwen/Qwen3.6-35B-A3B; chat_template.jinja is the canonical Qwen3.6 template. This repo adds the MTP graft, the lm_head re-quantization, and the template fix.

---

If you followed the upstream repo and got ~80 tok/s instead of 170

You were reading the wrong repo, and it is not obvious. sakamakismile/Ornith-1.0-35B-NVFP4 publishes only model.safetensors — no model-mtp.safetensors, no model.safetensors.index.json, no model-lmhead-w4.safetensors. Verified against the Hub API:

Without the index, vLLM loads a single file, the mtp.* tensors do not exist, and you get 1 token per forward pass instead of ~3.9 — no matter how correct your flags are.

This repo is the complete, ready-to-serve checkpoint. Point at it and use the serving command above. Nothing to assemble.

assemble.py is included only for the case where you want to graft the MTP onto your own copy of the upstream checkpoint instead of downloading this one. It reads safetensors headers (no 22 GB load) and refuses to emit an index that would not load.

Independent re-measurement, 2026-08-03

Re-measured on the same hardware with a different harness, discarding every sample taken while another request — or another pod on the same time-sliced GPU — was in flight:

Consistent with the 1K/8K figures above; the spread is workload, not variance. Always state the workload with a tok/s number — acceptance, and therefore throughput, swings by 2× between prose and code editing on the same engine.

Two measurement traps worth repeating, both of which cost us hours:

• A neighbouring pod on a time-sliced GPU silently halves your numbers. We measured the same config at 138.7 and at 26.8 tok/s. The contaminated figure looked entirely plausible. Gate on /metrics and discard samples taken with num_requests_running > 0.
• Verify speculation is actually on. curl /metrics | grep spec_decode_num_accepted_tokens_total. We once concluded speculative decoding "gave no speedup" when a kubectl apply had silently reverted the flag. "On but useless" and "never enabled" look identical from throughput alone.

Related

For qwen3_moe models (e.g. Qwen3-Coder-30B-A3B) this MTP module is not graftable — different architecture, dimensions and vocabulary. The equivalent there is an EAGLE3 draft head; complete measured recipe at pocharlies/Qwen3-Coder-30B-A3B-NVFP4-EAGLE3-DGXSpark.

---

Full benchmark campaign

Speed, tool-calling and agentic results for this model alongside three others on the same DGX Spark — including two optimizations that were measured and rejected — with all raw evidence:

https://huggingface.co/datasets/pocharlies/dgx-spark-moe-benchmarks