Collection • GitHub (MNN) • Tech Report

English | 中文

MNN-Pocket-0.3B

MNN-Pocket is a family of compact language models co-designed with the MNN inference stack for on-device deployment. MNN-Pocket-0.3B is the general-purpose, instruction-tuned member of the family.

• Tiny footprint: ~300M parameters; the INT4 (W4A16) QAT MNN build occupies only 173 MB.
• Fast edge inference: up to 466 tokens/s decode on iPad Pro M5 (Metal), above the 200 tok/s real-time target.
• Context length: up to 32K tokens.
• Balanced bilingual capability: best average (50.00) across 11 zh/en benchmarks among ~300M models, surpassing several ~500M baselines.

Demo

iPhone 18 Pro Demo

Your browser does not support the video tag.

Model Details

Architecture:

• Parameters: ~300M
• Layers: 28
• Hidden size: 896
• Attention: 14 heads, GQA with 2 KV heads, head_dim 64
• FFN intermediate: 2560 (SiLU)
• Context: 40,960 positions (32K supported in deployment)
• Vocabulary: 65,536
• Embeddings: tied input/output
• Precision: bfloat16 native; INT4 W4A16 (MNN build)
• Languages: Chinese, English

MNN-Pocket-0.3B is intended for on-device assistants, data extraction, and lightweight agentic pipelines. Like other sub-billion-parameter models, it is not recommended for reasoning-heavy workloads such as advanced mathematics or long-horizon code generation.

Chat Template

MNN-Pocket uses a ChatML-style conversation format. The exact template (including the im_start / im_end control tokens) ships in this repo as chat_template.jinja; prefer tokenizer.apply_chat_template() so formatting stays correct automatically. Schematic:

[IM_START]system
You are a helpful assistant.[IM_END]
[IM_START]user
Hello![IM_END]
[IM_START]assistant

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("taobao-mnn/MNN-Pocket-0.3B")
text = tok.apply_chat_template(
    [{"role": "system", "content": "You are a helpful assistant."},
     {"role": "user", "content": "Hello!"}],
    add_generation_prompt=True, tokenize=False)

Inference

MNN-Pocket is optimized for the MNN runtime but also runs in Transformers and llama.cpp.

On-device with MNN: the INT4 (W4A16, block 64) QAT build is published as MNN-Pocket-0.3B-MNN.

For other precisions, export from the safetensors in this repo with llmexport:

# Download the MNN source
git clone https://github.com/alibaba/MNN.git

# Export to MNN (INT4, block 64)
cd MNN/transformers/llm/export
python llmexport.py --path /path/to/MNN-Pocket-0.3B --export mnn --quant_bit 4 --quant_block 64

Build flags, config.json options and llmexport parameters are in the MNN-LLM doc.

With the MNN weights in hand — the published INT4 build or your own export — build the runtime and run it:

# Download the MNN source (only needed if you have not cloned it yet)
git clone https://github.com/alibaba/MNN.git

# Build
cd MNN
mkdir build && cd build
cmake .. -DMNN_LOW_MEMORY=true -DMNN_BUILD_LLM=true -DMNN_SUPPORT_TRANSFORMER_FUSE=true
make -j

# Run
./llm_demo /path/to/MNN-Pocket-0.3B-MNN/config.json prompt.txt

Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "taobao-mnn/MNN-Pocket-0.3B"
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "What is MNN?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                       return_tensors="pt", tokenize=True, return_dict=True).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True,
                         temperature=0.8, top_p=0.8, top_k=20)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

llama.cpp: see MNN-Pocket-0.3B-GGUF.

Performance

Post-trained MNN-Pocket-0.3B vs. same-size and larger open models (higher is better). Full protocol in the tech report.

The INT4 (W4A16) QAT build loses only 0.41 points on average (49.59 vs 50.00), confirming quantization largely preserves quality while cutting the footprint to ~173 MB.

On-Device Speed

Prefill / decode throughput at 512-token context with the INT4 build:

License

Released under the Apache License 2.0. See LICENSE.txt.

Citation

@misc{mnn2026pocket,
  title  = {MNN-Pocket-0.3B Technical Report},
  author = {{MNN Team}},
  year   = {2026}
}