Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill

This checkpoint is an experimental MTP-enabled derivative of shisa-ai/Ornith-1.0-35B-FP8-BLOCK. It keeps the original FP8_BLOCK / compressed-tensors Ornith base weights and adds a BF16 Qwen3.6 MTP head in model-mtp.safetensors. It starts from the same Qwen3.6 graft used for the official MTP checkpoint, shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP, then KL-distills only the MTP head.

The MTP recipe was adapted from protoLabsAI/Ornith-1.0-9B-MTP, which demonstrated grafting a same-family Qwen MTP head into an Ornith checkpoint and optionally KL-distilling only the MTP head. Thanks to protoLabsAI for the MTP graft/distillation technique and base recipe.

What Changed

• Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
• Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
• Added shard: model-mtp.safetensors
• Added tensors: 19 top-level mtp.* tensors
• MTP dtype: BF16
• Base weights: unchanged FP8_BLOCK / compressed-tensors
• Training: one KL-distill pass over a private local prompt/completion corpus, training only the MTP head while the Ornith verifier/base stayed frozen

The private training and benchmark prompts are not included in this repository because they were generated from personal rollouts and local repositories. The shareable recipe code is included under mtp_recipe/. The full local pipeline for this 5K run, including teacher generation and MTP KL training, completed within a few hours on a single RTX PRO 6000, so it should be practical to customize or improve for other workloads.

MTP Training

The KL-distill run trained only the MTP head. The Ornith base/verifier and shared lm_head stayed frozen.

Training setup:

• Hardware: one NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• Trainable parameters: 844.6M MTP-head parameters
• Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
• Initial MTP head: zero-training Qwen3.6 graft from Qwen/Qwen3.6-35B-A3B
• Output checkpoint: this model
• Objective: KL distribution matching against frozen Ornith verifier logits
• Epochs: 1
• Learning rate: 5e-5 with warmup ratio 0.05
• max_seq_len: 2048
• batch_tokens: 8192
• Gradient accumulation: 4 micro-batches per optimizer step
• Optimizer steps: 462

Timing:

• End-to-end distill command wall time: 71m 21s
• Model load and checkpoint write overhead: about 34s
• Approximate training-loop time: about 70-71m
• Approximate optimizer-step time: 9.3s/step

Corpus/token counts:

The training split intentionally preserved long-context examples, but this run used max_seq_len=2048. As a result, 2,650/4,500 train rows were skipped because the prompt left fewer than four completion tokens inside the training window.

Approximate throughput over the full distill wall time:

• Clipped forward tokens: ~2.92M tokens, or ~683 tok/s
• KL-scored target tokens: ~1.108M tokens, or ~259 tok/s

One implementation detail from this run: the script processed 1,850 usable examples, but only 1,848 contributed to optimizer updates because the final two examples were left over after accumulation by four. This is negligible for this artifact, but a future distill run should flush the final partial accumulation step.

Important Result Summary

This checkpoint did improve the offline KL proxy, but it was not the best serving result in our matched vLLM tests. The best measured row was still the official zero-training Qwen3.6 MTP checkpoint with num_speculative_tokens=3.

Use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP as the recommended throughput checkpoint from our testing. Use this distilled checkpoint as an experimental/diagnostic artifact unless your own serving tests show a win over that official MTP3 control.

Local Benchmark Methodology

Hardware:

• One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• Single-GPU serving on GPU0 only

Runtime:

• vLLM 0.23.0
• FlashInfer attention backend
• FP8 KV cache
• compressed-tensors quantization
• No LMCache for the benchmark rows below
• MAX_MODEL_LEN=32768
• MAX_NUM_SEQS=16
• MAX_BATCHED_TOKENS=32768
• MAX_CUDAGRAPH_CAPTURE_SIZE=16
• GPU_MEMORY_UTIL=0.95

Workload:

• Private custom validation benchmark derived from local code/agentic prompts
• 64 requests
• 63,327 total input tokens
• 16,384 generated tokens (256 per request)
• max_concurrency=1
• request_rate=inf
• temperature=0.6, top_p=0.95, top_k=20
• ignore_eos

The private benchmark data is not uploaded. The benchmark used vLLM's bench serve --dataset-name custom path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with prompt and output_tokens fields and run a command like:

CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp-qwen36-distill \
  --tokenizer /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Offline proxy on the same validation split:

• Official MTP, Qwen3.6 graft post hidden: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
• Qwen3.6 KL-distill post hidden: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Interpretation: the KL pass improved the offline distribution metric, but did not improve matched vLLM serving throughput versus the official MTP3 control.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

vllm serve /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --served-model-name ornith-35b-fp8-block-mtp-qwen36-distill \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

num_speculative_tokens=3 gave the best local throughput for this checkpoint, but it also reduced acceptance to about 67% and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default.

Recipe Code

The non-private code used for grafting, corpus generation, KL distillation, offline proxy evaluation, and validation is included in mtp_recipe/. It includes a sanitized example config at mtp_recipe/configs/ornith-35b-distill-qwen36.example.yaml.

The private corpus and private benchmark JSONL used in the local run are not included.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

• Source model: deepreinforce-ai/Ornith-1.0-35B
• Quantization tool: llm-compressor model-free PTQ
• Quantization format: compressed-tensors
• Scheme: FP8_BLOCK
• Calibration data: none; this is data-free/model-free PTQ
• Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
• Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
• Target modules: Linear
• compressed-tensors metadata version recorded in config.json: 0.15.1.a20260406

The quantization ignore list includes re:^mtp.*, so the MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

• Source model: deepreinforce-ai/Ornith-1.0-35B
• FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
• MTP donor: Qwen/Qwen3.6-35B-A3B
• MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}