English | 中文
MNN-Pocket-0.3B-MNN
MNN export of MNN-Pocket-0.3B for on-device inference on Android / iOS / HarmonyOS and edge devices with the MNN runtime.
MNN-Pocket is a family of compact (~300M parameter) language models co-designed with the MNN inference stack for on-device deployment. This repo ships the INT4 (W4A16) quantization-aware-training build — the format that actually goes into production apps.
- 173 MB total footprint
- QAT, not post-hoc quantization: INT4 average 49.59 vs 50.00 BF16 on the 11-benchmark suite (only 0.41 points down)
- Up to 466 tok/s decode on iPad Pro M5 (Metal), 251 tok/s on Snapdragon 8 Gen 5 (CPU) at 512-token context
- 32K context, ChatML-style chat template baked into
llm_config.json - Best for on-device assistants, data extraction and lightweight agentic pipelines; not recommended for reasoning-heavy workloads (advanced math, long-horizon code)
Find more details in the original model card: https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B
Demo
iPhone 18 Pro Demo
Your browser does not support the video tag.Model Details
| Property | Value |
|---|---|
| Parameters | ~300M |
| Precision | INT4 W4A16 (quant_bit 4, group size 64; tied-embedding alpha in fp16) |
| Weight size | 171 MB (llm.mnn.weight), ~173 MB total footprint |
| Architecture | Decoder-only transformer, 28 layers |
| Hidden size | 896 |
| Attention | 14 heads, GQA with 2 KV heads, head_dim 64 |
| FFN intermediate | 2560 (SiLU) |
| Context Length | 32K (40,960 position embeddings) |
| Vocabulary | 65,536 |
| Embeddings | tied input/output |
| Backends | CPU / Metal / OpenCL (set in config.json) |
| Languages | Chinese, English |
Files
| File | Size | Purpose |
|---|---|---|
llm.mnn |
0.95 MB | compute graph |
llm.mnn.weight |
171 MB | INT4 weights (quant_bit 4, group 64) |
llm_config.json |
3.2 KB | model layout + ChatML chat template + tied-embedding offsets |
tokenizer.mtok |
1.7 MB | MNN tokenizer |
config.json |
0.5 KB | runtime config: backend_type, thread_num, precision, memory, sampler |
🏃 How to run
# Download the MNN source
git clone https://github.com/alibaba/MNN.git
# Build
cd MNN
mkdir build && cd build
cmake .. -DMNN_LOW_MEMORY=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true
make -j
# Run
./llm_demo /path/to/MNN-Pocket-0.3B-MNN/config.json prompt.txt
The backend is chosen in config.json (backend_type: cpu / metal / opencl). Metal and OpenCL are off by default, so add -DMNN_METAL=ON / -DMNN_OPENCL=ON to the cmake line above if you need them.
Tuning the runtime is done in config.json — no re-export needed:
| Field | Shipped value | Notes |
|---|---|---|
backend_type |
cpu |
switch to metal / opencl for GPU backends |
thread_num |
4 | CPU threads; raise on devices with more performance cores |
precision / memory |
low / low |
favour speed and footprint; use normal to trade memory for accuracy |
| sampler | mixed, temp 0.8 | recommended generation settings: temperature 0.8, top_p 0.8, top_k 20 |
The INT4 tied-embedding layout requires a runtime from the same MNN revision family as the export; mismatched runtimes fail with
Load module failed.
Android / iOS / HarmonyOS integration follows the standard MNN LLM API; see the MNN repo for the mobile demos.
Chat Template
ChatML-style conversation format, shipped inside llm_config.json (the MNN runtime applies it
automatically). Schematic:
[IM_START]system
You are a helpful assistant.[IM_END]
[IM_START]user
Hello![IM_END]
[IM_START]assistant
📊 On-Device Speed (INT4, 512-token context)
| Device | Backend | Prefill (tok/s) | Decode (tok/s) |
|---|---|---|---|
| iPad Pro M5 | Metal | 8,721 | 466 |
| iPad Pro M5 | CPU | 2,228 | 231 |
| Snapdragon 8 Gen 5 | OpenCL | 3,403 | 138 |
| Snapdragon 8 Gen 5 | CPU | 930 | 251 |
📊 Benchmarks
INT4 (QAT) vs BF16 and same-size / larger open models (higher is better).
| Benchmark | INT4 (this repo) | BF16 | Gemma-3-270M | ERNIE-4.5-0.3B | LFM2.5-350M | SmolLM2-360M | Hunyuan-0.5B | MiniCPM4-0.5B | Qwen3-0.6B |
|---|---|---|---|---|---|---|---|---|---|
| Average | 49.59 | 50.00 | 6.42 | 31.99 | 31.28 | 18.43 | 31.22 | 46.81 | 44.28 |
| C-Eval | 40.27 | 38.86 | 6.84 | 32.17 | 32.10 | 23.77 | 35.29 | 67.98 | 44.06 |
| CMMLU | 39.65 | 40.11 | 4.94 | 33.91 | 31.65 | 24.87 | 35.69 | 66.04 | 44.50 |
| MMLU-Redux | 43.40 | 44.00 | 13.58 | 32.68 | 41.01 | 24.17 | 36.29 | 52.85 | 51.63 |
| BBH | 46.63 | 47.38 | 8.09 | 31.07 | 26.03 | 27.74 | 54.65 | 47.61 | 41.94 |
| IFEval | 74.12 | 73.20 | 20.15 | 44.92 | 72.09 | 32.16 | 24.58 | 52.13 | 56.93 |
| IFEval-zh | 72.49 | 74.59 | 17.02 | 32.87 | 44.06 | 25.64 | 25.87 | 30.30 | 50.35 |
| GSM8K | 52.84 | 54.36 | 0.00 | 31.24 | 33.66 | 11.60 | 36.01 | 42.46 | 45.49 |
| MGSM-zh | 48.80 | 51.20 | 0.00 | 25.20 | 18.00 | 3.60 | 34.80 | 37.60 | 40.00 |
| MATH500 | 35.40 | 35.40 | 0.00 | 21.40 | 14.80 | 4.60 | 35.80 | 23.00 | 45.20 |
| HumanEval | 53.66 | 53.66 | 0.00 | 38.41 | 14.63 | 0.00 | 14.63 | 48.17 | 36.59 |
| MBPP | 38.20 | 37.20 | 0.00 | 28.00 | 16.00 | 24.60 | 9.80 | 46.80 | 30.40 |
Other Formats
- MNN-Pocket-0.3B — BF16 safetensors, full model card (723 MB)
- MNN-Pocket-0.3B-GGUF — F16 / Q8_0 (BF16) and Q4_1 (QAT) for llama.cpp
- MNN-Pocket-0.3B-Base / MNN-Pocket-0.3B-SFT — pre-training / SFT checkpoints
- MNN-Pocket-0.3B-Dec — non-autoregressive decision model (LoRA adapter + pointer head)
License
Apache License 2.0. See LICENSE.txt.