中文Sign up

darkmaniac7/TokForge-AccelerationPack-Draft

Source: 🤗text-generationapache-2.01.2 GBother✓ 2 checksumsupdated 9d ago
or submit a torrent

Published by TokForge for on-device inference.

TokForge

Runs on-device in the TokForge app.

TokForge Acceleration Pack: Qwen3 KL-Distilled Draft Model

Overview

KL-distilled Qwen3-0.6B draft model for speculative decoding in TokForge. Trained on 10,000 teacher samples from Qwen3-8B.

What This Is

A small (0.6B) draft model that:

  • Runs on CPU alongside your main GPU model
  • Predicts tokens in parallel: main model batch-verifies and accepts correct ones
  • KL distillation matches the teacher's full logit distribution, not just top-1 tokens
  • KL distillation targets higher acceptance rates than stock or abliterated drafts

Results

Results are per device and per workload: backend routing, prompt length, and thermal state all change the outcome, and some device and target pairings do not benefit.

Why KL Distillation?

Approach How it trains Result
Stock (no training) Uses base weights Low acceptance, often regresses
Abliterated Removes refusal behavior Small acceptance gain, still limited
SFT (supervised) Trains on hard labels (top-1 token) Draft learns to copy text, not predict
KL Distillation Trains on full logit distribution Draft learns WHICH tokens are likely

KL divergence loss teaches the draft model to match the teacher's probability distribution across ALL tokens, not just the most likely one. This is critical because MNN's greedy sampler needs the draft's top-1 to match the teacher's top-1, and KL training optimizes exactly for this.

Training Details

  • Teacher: Qwen3-8B-HF (base)
  • Student: Qwen3-0.6B-HF (base, LoRA r=16, alpha=32)
  • Data: 10,000 teacher-generated samples (prose, code, Q&A)
  • Loss: 80% KL divergence + 20% cross-entropy, temperature=1.5
  • Training: 3 epochs, batch=4, grad_accum=4 (1,875 optimizer steps)
  • Final KL: 0.339 (21% lower than v1's 0.43 trained on 1K samples)
  • Hardware: 2x NVIDIA RTX PRO 6000 Blackwell (teacher GPU 0, student GPU 1)
  • Export: MNN Q4 quantization (quant_bit=4, quant_block=128)

Optimal Draft Config

Best with: target OpenCL, draft CPU, d=3, thread_num=2, power=high.

config_cpu.json:

{
    "backend_type": "cpu",
    "thread_num": 2,
    "precision": "low",
    "memory": "low",
    "sampler_type": "greedy",
    "power": "high"
}

Why thread_num: 2? On Android's WALT CPU governor, using too many threads (4+) can cause the scheduler to spread work across efficiency cores at low frequency. 2 threads stay on performance cores at high clock speeds.

Compatible Target Models

  • Qwen3-8B: supported
  • Qwen3-14B: supported
  • Qwen3-4B (MNN): Supported, enabled by default (CPU speculative decoding, fleet-verified on our test devices)
  • Qwen3.5: Not compatible (different architecture: LinearAttention vs full MHA)

SoC Compatibility

SoC GPU Notes
SM8850 (RedMagic/S26) Adreno 840 Primary targets
SM8635 (Xiaomi Pad 7 Pro) Adreno 735 Tested
SM8650 (S24/Lenovo) Adreno 750 Testing; may regress on 9B

Usage

Download via TokForge app: Settings > Spec Decode > Download Acceleration Pack

Typical TokForge recipe

  • target backend: opencl
  • draft backend: cpu
  • draft threads: 2
  • d=3
  • greedy sampling
  • low precision / low memory

Version History

Version Samples KL Date
v1 (abliterated) n/a n/a 2026-03-19
v2 (KL 1K) 1,024 0.43 2026-03-21
v3 (KL 10K) 10,000 0.339 2026-03-22

License: Apache 2.0 Source: KL-distilled from Qwen/Qwen3-0.6B using Qwen/Qwen3-8B as teacher Built with: TokForge

Limitations and Intended Use

  • Best current evidence is strongest on Qwen3-8B and Qwen3-14B.
  • Results depend heavily on backend routing, prompt length, and thermal state.
  • This is a TokForge-specific runtime bundle, not a standard Transformers checkpoint.

Credits

  • Qwen3-0.6B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
  • Qwen3-8B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license. Used as the distillation teacher.
  • Converted with llmexport from MNN by Alibaba (Apache-2.0).

Community