Published by TokForge for on-device inference.

TokForge

• Website: https://tokforge.ai
• Discord: https://discord.gg/Acv3CBtfVm
• Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
• iOS TestFlight: https://testflight.apple.com/join/jnufjzRr

Runs on-device in the TokForge app.

TokForge Acceleration Pack: Qwen3 KL-Distilled Draft Model

Overview

KL-distilled Qwen3-0.6B draft model for speculative decoding in TokForge. Trained on 10,000 teacher samples from Qwen3-8B.

What This Is

A small (0.6B) draft model that:

• Runs on CPU alongside your main GPU model
• Predicts tokens in parallel: main model batch-verifies and accepts correct ones
• KL distillation matches the teacher's full logit distribution, not just top-1 tokens
• KL distillation targets higher acceptance rates than stock or abliterated drafts

Results

Results are per device and per workload: backend routing, prompt length, and thermal state all change the outcome, and some device and target pairings do not benefit.

Why KL Distillation?

KL divergence loss teaches the draft model to match the teacher's probability distribution across ALL tokens, not just the most likely one. This is critical because MNN's greedy sampler needs the draft's top-1 to match the teacher's top-1, and KL training optimizes exactly for this.

Training Details

• Teacher: Qwen3-8B-HF (base)
• Student: Qwen3-0.6B-HF (base, LoRA r=16, alpha=32)
• Data: 10,000 teacher-generated samples (prose, code, Q&A)
• Loss: 80% KL divergence + 20% cross-entropy, temperature=1.5
• Training: 3 epochs, batch=4, grad_accum=4 (1,875 optimizer steps)
• Final KL: 0.339 (21% lower than v1's 0.43 trained on 1K samples)
• Hardware: 2x NVIDIA RTX PRO 6000 Blackwell (teacher GPU 0, student GPU 1)
• Export: MNN Q4 quantization (quant_bit=4, quant_block=128)

Optimal Draft Config

Best with: target OpenCL, draft CPU, d=3, thread_num=2, power=high.

config_cpu.json:

{
    "backend_type": "cpu",
    "thread_num": 2,
    "precision": "low",
    "memory": "low",
    "sampler_type": "greedy",
    "power": "high"
}

Why thread_num: 2? On Android's WALT CPU governor, using too many threads (4+) can cause the scheduler to spread work across efficiency cores at low frequency. 2 threads stay on performance cores at high clock speeds.

Compatible Target Models

• Qwen3-8B: supported
• Qwen3-14B: supported
• Qwen3-4B (MNN): Supported, enabled by default (CPU speculative decoding, fleet-verified on our test devices)
• Qwen3.5: Not compatible (different architecture: LinearAttention vs full MHA)

SoC Compatibility

Usage

Download via TokForge app: Settings > Spec Decode > Download Acceleration Pack

Typical TokForge recipe

• target backend: opencl
• draft backend: cpu
• draft threads: 2
• d=3
• greedy sampling
• low precision / low memory

Version History

---

License: Apache 2.0 Source: KL-distilled from Qwen/Qwen3-0.6B using Qwen/Qwen3-8B as teacher Built with: TokForge

Limitations and Intended Use

• Best current evidence is strongest on Qwen3-8B and Qwen3-14B.
• Results depend heavily on backend routing, prompt length, and thermal state.
• This is a TokForge-specific runtime bundle, not a standard Transformers checkpoint.

Credits

• Qwen3-0.6B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
• Qwen3-8B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license. Used as the distillation teacher.
• Converted with llmexport from MNN by Alibaba (Apache-2.0).

Community

• Website: tokforge.ai
• Discord: Join the Discord
• Collection: TokForge Mobile Draft Models