Published by TokForge for on-device inference.
TokForge
- Website: https://tokforge.ai
- Discord: https://discord.gg/Acv3CBtfVm
- Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
- iOS TestFlight: https://testflight.apple.com/join/jnufjzRr
Runs on-device in the TokForge app.
Qwen3-8B-abliterated-v2 (MNN)
Pre-converted Huihui-Qwen3-8B-abliterated-v2 (abliterated Qwen3-8B) in MNN format for on-device inference.
Model Details
- Architecture: Qwen3 (standard attention, 36 layers)
- Parameters: 8B (4-bit quantized)
- Format: MNN (Alibaba Mobile Neural Network)
- Vocab: 151,936 tokens
- Quantization: W4A16 (4-bit weights, 16-bit activations)
Files
| File | Size | Description |
|---|---|---|
llm.mnn |
631KB | Model graph |
llm.mnn.weight |
4.4GB | Quantized weights |
embeddings_bf16.bin |
1.2GB | BF16 embedding table (required) |
llm_config.json |
4.5KB | Model config with jinja chat template |
tokenizer.txt |
3.0MB | Tokenizer |
config.json |
210B | MNN runtime config |
Usage with TokForge
This model is optimized for TokForge, an Android app for on-device LLM inference.
Speculative decoding
It can be paired with a small CPU draft model for speculative decoding. Results vary by device and workload.
Draft model: Qwen3-0.6B
Abliteration
This model has been abliterated (safety filters removed) for unrestricted conversation. Use responsibly.
Limitations and Intended Use
- Intended for TokForge / MNN on-device inference, especially Android phones and tablets.
- It can be paired with a small CPU draft model for speculative decoding.
- Real throughput varies by SoC, thermal state, backend, and generation length.
- This repo is a runtime bundle, not a standard Transformers training checkpoint.
Credits
- Huihui-Qwen3-8B-abliterated-v2 by huihui-ai (Hugging Face). Used under the Apache-2.0 license. Our MNN files were converted from this model.
- Qwen3-8B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
- Converted with llmexport from MNN by Alibaba (Apache-2.0).
Community
- Website: tokforge.ai
- Discord: Join the Discord
Export
Converted using MNN's llmexport pipeline with --quant_bit 4 --quant_block 128.