Ornith-1.0-397B W4A16 + DFlash (vLLM, RTX Pro 6000)
Pilcothink's AutoRound W4A16 quant of Ornith-1.0-397B, packaged with the z-lab DFlash draft for speculative decoding. Tuned on 4× RTX Pro 6000 Blackwell (SM120, 4×96GB).
NOTE: This can likely be improved a lot further.
Usage
hf download fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash --local-dir Ornith-1.0-397B
cd Ornith-1.0-397B
./serve-rtx-pro.sh
OpenAI-compatible server on port 8000: TP4, 262144 context, reasoning parser qwen3, tool parser
qwen3_xml. Runs on voipmonitor/vllm:eldritch-enlightenment-v2226f26-b12x15cd38c-cu132-20260629
(vLLM 0.11.2.dev + native support for the DFlash sliding-window drafter). First run pulls the image
and downloads the draft weights. Override via PORT, TP, MODEL_DIR, DRAFT_DIR, IMAGE.
Config notes
--attention-backend FLASHINFER: split-KV decode keeps the multi-token verify step at ~20 ms per step regardless of context length.num_speculative_tokens=12: ~40% faster decode than k=8 at concurrency 1, equal at 2. k=16 (the draft's native block) is slower.- Keep the KV cache bf16 and
--async-schedulingoff; both alternatives lose speed and/or acceptance on this hybrid-GDN architecture. NCCL_P2P_LEVEL=SYS+VLLM_MARLIN_USE_ATOMIC_ADD=1: +13% decode, +40% prefill on 4× PCIe GPUs.
Benchmarks
4× RTX Pro 6000 Blackwell, TP4, streaming OpenAI requests, real coding prompts generated to natural EOS (≤2048 tokens), 3 repeats. Vanilla = same image and quant, default vLLM serving per the base model card, no speculative decoding. Accept = mean accepted tokens per verify step.
| prompt | conc | decode tok/s | vanilla tok/s | uplift | accept | TTFT p50 | ITL p50 |
|---|---|---|---|---|---|---|---|
| 1,000 | 1 | 197.7 | 121.2 | 1.63× | 4.20 | 152 ms | 21.0 ms |
| 10,000 | 1 | 139.5 | 119.9 | 1.16× | 3.71 | 1.28 s | 21.1 ms |
| 100,000 | 1 | 148.5 | 107.9 | 1.38× | 3.41 | 14.2 s | 22.6 ms |
| 1,000 | 3 | 149.1 | 80.1 | 1.86× | 4.20 | 392 ms | 29.2 ms |
| 10,000 | 3 | 104.7 | 75.6 | 1.38× | 3.45 | 2.86 s | 29.7 ms |
| 100,000 | 3 | 69.8 | 47.1 | 1.48× | 3.61 | 28.1 s | 33.6 ms |
Prefill throughput 6.7–8.7k tok/s.
Credits
- Model: DeepReinforce Team (MIT)
- Quant: Pilcothink, rehosted unmodified
- DFlash: z-lab (arXiv 2602.06036)
- vLLM image: voipmonitor eldritch builds
LocalMaxxing