Ornith-1.0-35B-Heretic-NVFP4-DFlash

NVFP4 build of Ornith-1.0-35B-A3B set up for DFlash speculative decoding in vLLM. The point of this repo is not the weights — it is the recipe: how to get NVFP4 + DFlash + FP8 KV + long context + tensor-parallel all running at the same time on Blackwell hardware that has no native FP4 (GB10 / sm_121 and dual RTX 5070 Ti / sm_120).

On 2× 16 GB the KV pool measures ~347k tokens — a 35B agent model with room for up to ~256k single-stream context. Everything here is benchmarked and verified at 128k (256k is within the pool budget, not yet load-tested end to end).

That combination is fiddly. Getting each piece alone is easy; getting all four at once has exactly one thing that makes it work, and a pile of dead ends. This repo documents both.

• Base model: Ornith-1.0-35B-A3B (Qwen3.6-35B-A3B family), NVFP4 (modelopt), MIT
• Draft model: z-lab/Qwen3.6-35B-A3B-DFlash — attached at serve time, not re-uploaded here (Apache-2.0)
• Engine: vllm/vllm-openai:nightly — the one image where this works (see below)
• Sibling repo (MTP variant): pottokao/Ornith-1.0-35B-Heretic-NVFP4-MTP — same model with a trainable MTP head instead of DFlash

---

TL;DR — the one thing that matters

The engine image. DFlash on a no-native-FP4 Blackwell card only comes together on a vLLM nightly build:

Pin the digest (nightly moves):

# verified on GB10 / sm_121
vllm/vllm-openai@sha256:5bda7078b1bb17f74d369e3ded63115a77d5ea5eeb9eab6ca9a52d108f9a262d   # vLLM 0.23.1rc1.dev968+g2c17d33f4
# verified on dual RTX 5070 Ti / sm_120 (TP2)
vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9   # vLLM 0.23.1rc1.dev1000+g95ed0feaa

---

Quick start

GB10 / DGX Spark (sm_121, ~121 GB unified) — memory free, path is the wall

Two separate things, and only the memory is easy:

• Memory: nothing to optimise. Set a high utilisation and go; pick your own context / concurrency. The memory tricks further down are a 16 GB problem GB10 doesn't have.
• Path: the same image gate from the top — and sm_121 is where it bites hardest. muse-glimmer has no sm_121 kernels (CUDA assert, sm_121-specific); aeon forces FlashInfer (Window-left crash). Use the nightly sm_121 digest, and keep the backends exactly: TRITON_ATTN (main and in speculative-config), --moe-backend humming, FP8 KV. sm_121 also has no native FP4 — dense NVFP4 falls to Marlin (weight-only), expected; don't hunt for a faster FP4 path that doesn't exist.

We verified only that NVFP4 + DFlash runs clean and fits on GB10 — we did not tune it.

docker run -d --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/Ornith-DFlash:/model \
  -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai:nightly \
  serve /model --trust-remote-code \
    --quantization modelopt --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
    --max-model-len 131072 --gpu-memory-utilization 0.85 \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":4,"attention_backend":"TRITON_ATTN"}'

Dual RTX 5070 Ti (sm_120, 16 GB × 2, TP2) — the tight recipe

This is where every flag earns its place. Working config, 128k context + cudagraph:

docker run -d --gpus all \
  -e NCCL_P2P_DISABLE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/Ornith-DFlash:/model \
  -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai:nightly \
  serve /model --trust-remote-code --tensor-parallel-size 2 \
    --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --moe-backend humming \
    --gpu-memory-utilization 0.95 --kv-cache-memory 3200000000 --max-model-len 131072 \
    --max-num-seqs 2 --max-num-batched-tokens 2048 --enable-chunked-prefill \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":4,"moe_backend":"humming","attention_backend":"TRITON_ATTN"}'

---

Backends — what must be set, and why (16 GB TP2)

Each of these, if wrong, either crashes or silently halves your speed:

---

Memory model (the part that surprised us)

On the GDN/mamba hybrid, the KV pool size is set by --kv-cache-memory, not by --max-num-seqs.

--max-num-seqs 2  →  KV pool 347,124 tokens,  free 561 MiB/card
--max-num-seqs 4  →  KV pool 347,124 tokens,  free 359 MiB/card   (same pool!)

max-num-seqs is almost free — each concurrent sequence just reserves one mamba-state slot (~100 MiB/card). So the real picture is a single shared budget you split three ways:

The ~347k-token pool → spend it on whichever you need:

• context — up to ~256k single-stream (--max-model-len)
• concurrency — more --max-num-seqs (nearly free, ~100 MiB/card each)
• speculation depth — raise num_speculative_tokens (K) 4→6+ for longer accept length

context × concurrency ≤ pool. Want a bigger pool → raise --kv-cache-memory until free memory bottoms out. It's very flexible — tune to your workload.

The memory hog is not DFlash's draft (737 MB) — it is the target model's own GDN/mamba linear-attention state (30 of 40 layers), which is per-sequence and does not shard across TP ranks (Setting attention block size to 2160 tokens to ensure page size ≥ mamba page size).

128k / cudagraph / concurrency — pick your point (all measured, TP2 16 GB × 2)

The pool holds ~347k tokens, i.e. roughly 256k of usable single-stream context if you want it. What to do with the headroom over 128k is your call — spend it on longer context (up to ~256k) or on more speculation (num_speculative_tokens 4 → 6+). z-lab suggests block size 8 (throughput) or 16 (single-user accept length); 4 here is deliberately conservative.

Note: with speculative decoding, higher concurrency does not always mean more throughput — rejected drafts burn verify compute, so AGG can fall as you add streams. Measure your own load.

---

Benchmarks — DFlash vs MTP, both platforms

Same harness both platforms: vllm bench serve (Spec-Bench), concurrency 1, 20 warmups, --ignore-eos, enable_thinking:false, K=4. mine/* = held-out agent-code (the judges); spec/* = Spec-Bench guard categories (regression check). MTP baseline = the sibling MTP repo.

tok/s is a property of the hardware; acceptance length (acc) is a property of the model. Read both.

Dual RTX 5070 Ti (sm_120, TP2, 16 GB × 2)

Dead even on speed (DFlash marginally ahead on the two agent judges); acceptance a hair lower for DFlash. DFlash's real edge here is the KV pool + 128k, which MTP can't match on 16 GB.

GB10 / DGX Spark (sm_121, single card)

On a single card MTP is marginally ahead on most categories (DFlash ~93–98% of MTP tok/s, acc a touch lower), except math/coding where DFlash edges up. Essentially tied — which matches the folk wisdom: DFlash ≈ MTP on speed, slightly behind on acceptance, and it buys you the context.

---

DFlash vs MTP — when to use which

Not a knockout — a fit question.

Verdict on 16 GB TP2: DFlash ties MTP on speed (slightly ahead on the agent-code judges), is marginally lower on acceptance, and gives a much larger KV pool + 128k. Since KV is gold for agents (long tool histories, big system prompts), DFlash is the better no-training default here. MTP still wins if you invest in training and don't need the context.

---

Compatible, but unverified

The recipe is a serve-time attachment for the Qwen3.6-35B-A3B family — any model on that base can load the same z-lab draft (sakamakismile/Qwen-AgentWorld-35B-A3B, huihui variants, …).

Qwen-AgentWorld-35B-A3B is a natural target (headless, agent-trained, 262k context window). But treat these as variables to test, not claims:

• The z-lab draft was trained against the base distribution; on an agent-finetuned model the acceptance rate may drop — measure it.
• AgentWorld ships as compressed-tensors W4A4 (not modelopt) and is multimodal (vision tower in full precision) → the flags and the memory budget differ. Untested.

---

Credits & licenses

• Ornith-1.0-35B-A3B — base model (MIT)
• z-lab/Qwen3.6-35B-A3B-DFlash — the draft model that does all the speculation (Apache-2.0). All credit for DFlash to z-lab.
• vLLM — engine (Apache-2.0); the nightly build is what makes this possible.

MIT (base) + Apache-2.0 (draft) — compatible. This repo adds no new model; it packages a base and a run recipe.