LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 10.1 tok/s · prefill 206 tok/s · TTFT 1.18 s · all 1243 ops delegated (2026-08-24); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 12.0 tok/s · prefill 84 tok/s · TTFT 540 ms · all 1243 ops delegated (2026-08-19); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 29.7 tok/s · prefill 106 tok/s · TTFT 2.14 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen25-coder-1.5b-int4/CARD.md

Qwen2.5-Coder-1.5B-Instruct — LiteRT-LM

Try it in your browser: the LiteRT-LM Web chat demo loads Qwen2.5-Coder-1.5B-Instruct_int4.litertlm (1.12 GB download) and runs it with WebGPU.

Qwen/Qwen2.5-Coder-1.5B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.16.

A 1.54B code model at 1.12 GB — small enough that a phone-class device runs it at conversational speed rather than merely running it. Code assistance is the use case that most needs that: it returns short outputs constantly, so decode rate is the experience.

A 3B sibling exists in this org — litert-community/Qwen2.5-Coder-3B-Instruct, 3.43 GB, converted by another contributor. Pick by what you are optimising: that one for capability, this one for speed and footprint. The two have not been benchmarked against each other on the same machine, so this card makes no claim about their relative quality.

Correctness

Two gates, because a general-knowledge check certifies nothing for a code model:

• 8-question sanity gate: 8/8 on the GPU backend (Mac, litert-lm 0.16.0).
• Code gate: 6/6. Six small functions — fib, reverse_words, is_prime, largest contiguous sublist sum, count_vowels, flatten — where the generated code is executed against assertions, not read. A task counts only if the code imports and every assertion holds.

Usage

litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm \
  --prompt "Write a Python function that returns the nth Fibonacci number. Code only."

# GPU
litert-lm run ./Qwen2.5-Coder-1.5B-Instruct_int4.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Qwen's ChatML markers including the vendor's default system prompt (You are Qwen, created by Alibaba Cloud. You are a helpful assistant.), which the upstream template inserts whenever no system message is supplied — rendering byte-identical to upstream for a single-turn request. A 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1).

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --cache no, quiet machine, serialized:

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

For scale, a 3B-class int4 bundle converted on the same day and measured on the same machine and protocol runs 1241 tok/s prefill / 86.3 decode / 0.233 s TTFT. Decode on these runtimes is memory-bandwidth-bound, so halving the weights is the most direct speed lever there is.

Honest notes:

• A Mac's GPU-vs-CPU ratio does not transfer to a phone. Here the GPU decodes 2.9× the CPU; on a Pixel 8a a 3B model's GPU and CPU decode land within 1% of each other, because a phone's CPU and GPU share the same LPDDR. On phone hardware the GPU's win is prefill and time-to-first-token.
• The embedding table is stored in its own bundle section. That is what keeps this file at 1.12 GB — with the table quantized inline it duplicates per prefill signature and the same weights come out at 2.53 GB.

Galaxy S26 — GPU and CPU

The published bundle runs on the Android GPU backend and generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_1024; the runtime accepts that split.

Speed on the same handset with one binary and one 205-token prompt file; only --backend changes (litert_lm_advanced_main from litert-lm 0.16.0, --benchmark, --sampler_backend=cpu). The GPU row is the gate run of 2026-08-24 (one run); the CPU rows were taken 2026-09-05, two cold runs (compile and weight caches deleted between runs) with the phone cooled below 42 °C before each run.

The CPU decodes 3.3× faster; the GPU wins prefill and peaks lower. A code answer is mostly decode (the bench replies ran 360–409 tokens), so this repo's litertlm_manifest.json recommends cpu for flagship-class Android on that pair. Pick gpu for long prompts with short answers.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0 — a pristine released stack, no patched checkout.

• EXTERNALIZE_EMBEDDER=1 is required, not cosmetic. Qwen2.5-Coder ties its embedding and lm_head, so a recipe asking for int4 linears and an int8 embedder describes one tensor two ways; the quantizer resolves it by copying the 151936×1536 table once per signature. Seven copies, 1.63 GB, 65% of the file — and no error anywhere. Externalising the table removes the conflict.
• The default system prompt is baked into the template. Upstream inserts it when the caller sends no system message; a plain ChatML template would drop it and put the model in a state it was not tuned in on every default request.
• Quantization: int4 blockwise-32 with OCTAV clipping on the linears, int8 on the embedding.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

Known runtime issues

External weights (since 2026-10-30). In Qwen2.5-Coder-1.5B-Instruct_int4.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier file with the weights inside the graph stays at revision 7840209.

From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings (read in the source of LiteRT-LM 0.16.0 to 0.18.1). Measured on this file with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) on the Mac CPU, greedy: after "Remember this number: 4182.", the answer to "What number did I ask you to remember?" was "I asked you to remember the number 4182.", the text this file's own graphs give with the stop token's row set to zero; with the token's real row they give "You asked me to remember the number 4182." Three other two-turn conversations gave the same answer either way. One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: this file carries stop sequences of <|im_end|> and a newline with each of 13 characters in front (12 punctuation marks and a space), so the runtime holds the mark back to see whether <|im_end|> and a newline follow, the answer ends on <|im_end|> alone, and the held mark is never written. LiteRT-LM 0.17.0 and later write the held mark (StopTokenStreamFilter in runtime/core/tasks.cc, read in the source of 0.17.0 to 0.18.1). Not measured on this file: this follows from the runtime's source read with this file's stop list and tokenizer; the drop itself was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle whose stop sequences have the same form.

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle. This repository is a community conversion and is not affiliated with Alibaba Cloud or the Qwen team.