Ornith-1.0-35B-Heretic-NVFP4-MTP

Ornith-1.0-35B-A3B · Heretic-abliterated · NVFP4 · MTP v2 (4-layer, K=4)

A derivative of deepreinforce-ai/Ornith-1.0-35B — a Qwen3.5-MoE 35B-A3B agentic-coding model — repackaged for fast local inference: NVFP4-quantized, fitted with a self-trained 4-layer Multi-Token-Prediction (MTP) head for speculative decoding, and Heretic-abliterated (uncensored). This is the second-generation MTP build: the draft head was retrained from scratch specifically for agentic-coding workloads (agent traces + code-heavy distillation) with a corrected LR schedule, and it beats the v1 head across the board on offline validation. The MTP head is this repo's own work — it is not part of the upstream Ornith release.

All credit for the base model goes to DeepReinforce AI (MIT-licensed). This repo only adds NVFP4 quantization, the MTP draft head, and abliteration on top.

On its trained target — agentic-coding / tool-calling — the MTP head accepts ~3.1 of 4 draft tokens per step (per-position 92/83/75/67%), pushing single-stream decode to ~370 tok/s single-shot (peaks ~405), and ~440 tok/s sustained in warm multi-turn on 2× RTX 5070 Ti (TP2). On free prose the head barely helps — see Performance below for the honest content-by-content picture.

ℹ️ Text-only. The base is a natively multimodal checkpoint, but the vision tower is not shipped/loaded here — this model is served in text-only mode.

---

Model specs

---

What changed in MTP v2

The draft (MTP) head was retrained from a v2 warm start:

• Data: code-primary distillation set + agent traces (v1 was more general-purpose).
• Training fix: corrected LR-schedule units.
• Architecture: 4 NVFP4-quantized MTP layers; experts in NVFP4, the rest BF16.
• vLLM's native spec_step_idx % mtp_num_hidden_layers routing means each speculative position uses its own dedicated layer (mtp_num_hidden_layers = 4), zero patching.

Offline validation (per-position acceptance, layers 0→3):

Performance — where the MTP head pays off

Measured single-stream on 2× RTX 5070 Ti (32 GB total, TP2), K=4, humming backend, via the repo's frozen probe (mtp_probe.py, which logs every run — reproducible with one command). The head is trained for agentic-coding / structured output, and that's where the speedup lands.

Agent / tool-calling (the trained target) — per-position acceptance at K=4:

→ ~3.1 of 4 draft tokens accepted per step (~4 tokens generated per forward pass). Even the 4th position clears ~67%, so all four layers earn their slot — this is why K=4, not K=2/K=3. Single-stream ~370 tok/s, peaks ~405.

Compare accepted-tokens-per-step across configs, not aggregate acceptance % — a K=2 head only drafts the two easy leading positions, so its average % looks higher while producing fewer tokens per step.

Acceptance is content-dependent (open-ended generation jitters run-to-run; fixed-format is tight):

Runs entirely on two consumer 16 GB cards — no datacenter GPU needed.

Warm multi-turn (real agent-loop usage). The per-content tok/s above are single-shot (one cold request). In a sustained conversation — prefix cache warm, context growing — steady-state decode reaches ~440–450 tok/s across content (canon.py, 5-turn), the more representative figure for multi-turn agentic work.

What the MTP head was (and wasn't) trained for. The draft head is purpose-trained on agentic-coding — agent traces, tool-calling, and code. It was not trained on free prose, creative writing, or open-ended chat; on those the draft acceptance drops to ~30% and you get little speculative speedup (decode drops to ~200 tok/s). Note this affects speed only — the base Ornith model still answers prose/chat at full quality, you just don't get the MTP acceleration there.

---

Internal quality/speed A/B

⚠️ This is an internal 60-prompt A/B (code / hard-reasoning / mixed, 20 each), self-assessed and single-stream — it is not a standard public benchmark. Reported for transparency, not as a leaderboard claim.

Takeaway: the A3B MoE decodes ~2.5× faster than a comparable dense model at near-equal answer quality, because only ~3B params are active per token.

---

Benchmark — ASUS GX10 (GB10), second platform

Measured on NVIDIA GB10 (unified 121 GiB LPDDR5X, single device, TP1), batch-1 single-stream, K=4, FP8 KV cache, humming MoE backend, vLLM nightly. Harness: Spec-Bench (gatekeeper) + held-out agent-code (main judge). Output is lossless (target-verified).

Compared on the same GB10 box against a DFlash baseline — AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4

• z-lab/Qwen3.6-35B-A3B-DFlash (K=11, BF16 KV).

• acc-rate = accepted / drafted per step ≈ (acc-len − 1) / K. The MTP head accepts ~58% of its 4 drafts vs ~32% for DFlash's 11 — nearly 2× the per-draft efficiency: it drafts a few high-confidence tokens instead of casting a wide net.
• acc-len (tokens/step) favours DFlash's deeper draft, but on a single bandwidth-bound GB10 that depth converts to only ~6% average throughput.
• Net: comparable single-stream speed at 1/3 the draft width and half the KV memory (FP8) — precision over volume. On the trained agent target, first-position acceptance stays 90%+.

Not a single-variable A/B — base model, speculative method, and KV dtype (FP8 vs BF16) all differ. GB10 is a single-device part; the 2× RTX 5070 Ti (TP2) numbers above are the high-bandwidth reference.

Benchmark — MTP vs DFlash, controlled 27B comparison (2× RTX 5070 Ti)

Ornith's 35B-A3B has no same-size public DFlash draft, so the cleanest MTP-vs-DFlash test is run on our 27B dense sibling (same base family, same rig, same harness). Hardware: 2× RTX 5070 Ti (16 GB each = 32 GB, TP2). All three use FP8 KV.

• Speed is a three-way tie (~85–89 tok/s). DFlash's deeper acceptance (acc-len 4.93 vs 3.2–3.6) does not convert to throughput — its 15-token draft cost cancels the gain.
• MTP K=3 is fastest and leanest: a 3-token draft at 73% acceptance vs DFlash's 26%. Precision beats casting a wide net (acc-rate = accepted / drafted ≈ (acc-len − 1) / K).
• Context is the real gap — MTP 96–128k vs DFlash 8k. DFlash needs a separate draft model that consumes VRAM, so on 2× 16 GB the context window stays small (~8k benched, ~32k practical ceiling). The MTP head is grafted into the target (near-zero extra VRAM), so it keeps the full 96–128k window.
• Net on memory-constrained multi-GPU: MTP matches DFlash speed at 1/5 the draft width, ~3× the per-draft efficiency, and 3–16× the usable context (128k vs ~8–32k).

Why an external benchmark?

The MTP draft head is trained only on an agent-coding / tool-calling distribution. Crucially, this can affect exactly one thing: the draft acceptance rate — how often the head's guesses match the target, i.e. the speedup. It can never alter the model's output: speculative decoding is lossless, the target verifies every token, and answers are identical whether the head is accurate or not. Our concern was narrow but real — could specializing the head on agent-code severely degrade its acceptance rate on task types it was never trained on (translation, summarization, QA, math, RAG, coding)? To test exactly that, we add Spec-Bench, an external suite of untrained task types, as a regression gate on acceptance. It guards speed generalization only; output quality is unaffected by construction and out of scope.

Usage (vLLM)

Requires a vLLM build that supports qwen3_5_moe + MTP speculative decoding + NVFP4 MoE. NVFP4 MoE is not accepted by the triton backend — you must use the humming MoE backend.

# 1. Download the weights from the Hub
hf download pottokao/Ornith-1.0-35B-Heretic-NVFP4-MTP --local-dir ./ornith-heretic

# 2. Serve with vLLM (2× 16 GB GPU, TP2)
docker run -d --name ornith --gpus all \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=134217728 \
  -v $(pwd)/ornith-heretic:/model \
  -p 8000:8000 \
  <your-vllm-image> \
    --host 0.0.0.0 --port 8000 \
    --model /model \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --moe-backend humming \
    --attention-backend TRITON_ATTN \
    --gpu-memory-utilization 0.92 \
    --max-model-len 131072 \
    --max-num-seqs 12 \
    --max-num-batched-tokens 4096 \
    --enable-chunked-prefill --enable-prefix-caching --async-scheduling \
    --speculative-config '{"method":"mtp","num_speculative_tokens":4,"moe_backend":"humming","attention_backend":"TRITON_ATTN"}' \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml --enable-auto-tool-choice \
    --default-chat-template-kwargs '{"enable_thinking": false}'

Key flags (why each matters)

Notes:

• Recommended K=4. The 4-layer head is trained for it; running at K<4 wastes the extra layers, and the head is tuned for agentic-coding acceptance rates at K=4.
• Memory / context: on 32 GB total VRAM (2× 16 GB, TP2) you can serve ~128K KV context (--max-model-len 131072). The base architecture supports 256K positions, but 128K is the practical KV budget at this VRAM after NVFP4 weights (~24 GB).
• Tested TP2 on 2× RTX 5070 Ti (16 GB each). --max-num-seqs 12 is tuned for that VRAM; raise it if you have more headroom.

---

Limitations & responsible use

• Uncensored: safety alignment has been removed (abliteration). This model will follow instructions the original refused. You are responsible for how you deploy it and for complying with applicable laws and the base model's license/usage policy.
• Text-only: the vision/video capability of the base checkpoint is not available here.
• Serving-specific: the MTP head needs the humming MoE backend and an NVFP4-aware vLLM; it will not run on stock/older engines without those.
• Benchmarks above are self-run and single-stream; treat concurrency/throughput numbers as untested unless you measure them yourself.

Acknowledgements

• Base model: deepreinforce-ai/Ornith-1.0-35B by DeepReinforce AI (MIT). Post-trained on Qwen3.5 (Alibaba Qwen team). All base-model credit is theirs; this repo is a downstream inference-optimized derivative.
• Abliteration: Heretic by Philipp Emanuel Weidmann.
• Quantization: NVIDIA TensorRT Model Optimizer (modelopt 0.45.0).
• MTP head trained and grafted in-house.

License

MIT, inherited from the base model. Redistribution keeps the MIT notice and attribution to DeepReinforce AI.