Sign up

litert-community/Qwen2-0.5B-Instruct

🤗 On Hugging Facetext-generationapache-2.0647 MBother✓ HF checksums availableupdated today
Magnet

Qwen2-0.5B-Instruct LiteRT-LM Model

This repository contains LiteRT-LM variants of Qwen/Qwen2-0.5B-Instruct optimized for on-device text generation.

Available Artifact

| File | Quantization Recipe | Context | Size |

|---|---|---:|---:|

| Qwen2_0.5B_Instruct.litertlm | dynamic_wi8_afp32 | - | 647.4 MB |

Performance (on device, measured)

Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL) against CPU (XNNPACK), one fixed 205-token prompt text (213 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the file was run on each backend with a real prompt and both produced a correct text answer; the GPU rows are full delegation of the transformer graphs (decode 1063/1063 and prefill 972/972 ops on LITERT_CL; only the tiny embedding-lookup helper graphs stay on the CPU).

| Backend | Prefill (213 tok) | Decode | Time-to-first-token | Init | Peak RSS |

|---|---|---|---|---|---|

| GPU (OpenCL) | 1259–1452 tok/s | 61.9–62.9 tok/s | 0.16–0.19 s | 0.8 s | 559–561 MB |

| CPU (XNNPACK) | 405–528 tok/s | 49.1–54.5 tok/s | 0.42–0.55 s | 0.7–1.0 s | 1005–1044 MB |

What the table says:

  • The GPU leads everywhere on this bundle: prefill 2.7–3.6×, decode ~1.2×, and peak RSS 1.8× lower (560 against ~1020 MB) — for a 0.5B int8 model that decode margin is unusual; most small bundles tie the CPU on decode.
  • GPU engine init is 0.8 s — unusually cheap for the GPU path (most bundles pay several seconds), so the GPU backend costs almost nothing extra even for short-lived processes.

Integration

Ready to integrate this into your product? Get started in the LiteRT-LM documentation.