English | 中文
MNN-Pocket-0.3B-DFlash2
DFlash2 speculative-decoding draft model for MNN-Pocket-0.3B, served on device by the MNN speculative-decoding runtime. Instead of proposing one token at a time, it generates a whole block of B = 8 tokens in a single draft forward pass, which the target model then verifies in one pass.
- 1.65x average decode speedup on iPad Pro (M5) and 1.15x on Snapdragon 8 Gen 5 against the INT4 MNN target
- Up to 2.06x on code, 1.63x on math, 1.26x on chat (iPad Pro M5)
- Mean accepted length ~4.6 tokens per decoding round
- Tiny draft: 3 layers, ~130 MB in BF16 — it shares the target's token embedding and LM head, so no vocabulary projection is duplicated
- Trained with SpecForge on Open-PerfectBlend prompts plus 500K post-training samples, using the position-weighted loss of D-PACE
Find more details in the original model card: https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B
Demo
iPhone 18 Pro Demo
Your browser does not support the video tag.Model Details
| Property | Value |
|---|---|
| Role | Speculative-decoding draft (not a standalone chat model) |
| Architecture | DFlash2DraftModel, decoder-only, 3 layers |
| Hidden size | 896 |
| Attention | 14 heads, GQA with 2 KV heads, head_dim 64 |
| FFN intermediate | 2560 (SiLU) |
| Draft vocabulary | 65,536 (embedding + LM head shared with the target) |
| Block size | 8 tokens per draft pass |
| Target model | MNN-Pocket-0.3B (28 layers); taps target_layer_ids [1, 7, 13, 19, 25] |
| Draft config | conv_kernel_size 2, conv_group_size 16, selector_rank 256, selector_top_k 16, mask_token_id 65521 |
| Context Length | 40,960 position embeddings |
| Precision | bfloat16 |
| Size | ~130 MB (model.safetensors) |
Files
| File | Size | Purpose |
|---|---|---|
config.json |
1.3 KB | draft architecture + dflash_config |
model.safetensors |
130 MB | draft weights (BF16) |
No tokenizer is shipped: the draft reuses the tokenizer of the target model (MNN-Pocket-0.3B).
🏃 How to run
The draft is driven by the MNN speculative-decoding runtime together with the target model
(MNN-Pocket-0.3B-MNN); it is not meant to
generate text on its own. Reference configuration used for the numbers below: block size B = 8,
batch size 1, greedy decoding, max generation length 256.
📊 Mean Accepted Length (draft x target configurations)
Mean accepted tokens per decoding round across six prompt-only benchmarks (higher is better). "MNN" = full-precision target, "MNN-INT4" = QAT INT4 target.
| Draft precision | Training source | Verification model | MATH-500 | AIME25 | MBPP | HumanEval | MT-Bench | Arena-Hard | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| BF16 | MNN | MNN | 5.34 | 4.59 | 5.55 | 5.83 | 3.55 | 3.18 | 4.67 |
| BF16 | MNN | MNN-INT4 | 5.30 | 4.49 | 5.57 | 5.75 | 3.62 | 3.16 | 4.65 |
| BF16 | MNN-INT4 | MNN-INT4 | 5.34 | 4.55 | 5.64 | 5.82 | 3.70 | 3.18 | 4.70 |
| INT4 | MNN | MNN-INT4 | 5.16 | 4.37 | 5.48 | 5.65 | 3.50 | 3.06 | 4.54 |
| INT4 | MNN-INT4 | MNN-INT4 | 5.22 | 4.47 | 5.55 | 5.76 | 3.65 | 3.10 | 4.63 |
Training the draft on the INT4 (QAT) target recovers most of the acceptance lost by verifying with a quantized target: under an MNN-INT4 verifier the MNN-trained draft trails the MNN-INT4-trained draft by 1.06% in BF16 and 1.94% in INT4.
📊 On-Device Speedup
Release configuration: MNN-Pocket-0.3B MNN-INT4 target with the INT4 MNN-trained draft (AIME25 / HumanEval / MT-Bench, max generation length 256).
| Device | Workload | AR tok/s | Mean AL | Spec. tok/s | Speedup |
|---|---|---|---|---|---|
| iPad Pro (M5) | Math | 435 | 4.49 | 709 | 1.63x |
| iPad Pro (M5) | Code | 435 | 5.73 | 895 | 2.06x |
| iPad Pro (M5) | Chat | 435 | 3.51 | 547 | 1.26x |
| iPad Pro (M5) | Avg. | 435 | 4.57 | 717 | 1.65x |
| Android (8 Gen 5) | Math | 135 | 4.48 | 159 | 1.17x |
| Android (8 Gen 5) | Code | 135 | 5.72 | 186 | 1.38x |
| Android (8 Gen 5) | Chat | 135 | 3.49 | 119 | 0.88x |
| Android (8 Gen 5) | Avg. | 135 | 4.56 | 155 | 1.15x |
Math and code workloads benefit more than chat. Speedup is measured against autoregressive decoding with the same target, backend and workload; similar accepted lengths can still yield different speedups depending on device and backend.
Other Formats
- MNN-Pocket-0.3B — BF16 target model, full model card (723 MB)
- MNN-Pocket-0.3B-MNN — INT4 W4A16 QAT target for on-device deployment (173 MB)
- MNN-Pocket-0.3B-GGUF — F16 / Q8_0 (BF16) and Q4_1 (QAT) for llama.cpp
- MNN-Pocket-0.3B-Base / MNN-Pocket-0.3B-SFT — pre-training / SFT checkpoints
- MNN-Pocket-0.3B-Dec — non-autoregressive decision model (LoRA adapter + pointer head)
License
Apache License 2.0. See LICENSE.txt.