LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, qwen3-4b-thinking): Mac Studio M4 Max · LiteRT-LM 0.14.0 · GPU · decode 68.3 tok/s · prefill 970 tok/s · TTFT 279 ms (2026-07-23); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 6.0 tok/s · prefill 63 tok/s · TTFT 3.40 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.5 tok/s · prefill 11 tok/s · TTFT 25.49 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen3-4b-thinking/CARD.md

Measured on device (edge-compat, qwen3-4b-thinking-dynamic-wi4b32-afp32): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 11.2 tok/s · prefill 129 tok/s · TTFT 1.65 s · all 1282 ops delegated (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.4 tok/s · prefill 9 tok/s · TTFT 29.40 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen3-4b-thinking-dynamic-wi4b32-afp32/CARD.md

Qwen3-4B-Thinking-2507 — LiteRT-LM (blockwise int4)

Qwen/Qwen3-4B-Thinking-2507 converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Qwen3-4B-Thinking-2507 is a dense 4B reasoning model (Qwen3ForCausalLM, 36 layers) that operates exclusively in thinking mode — it emits a <think>…</think> chain before its answer — so it rides the existing Qwen3 converter and runtime directly.

Artifact (Block 128)

Artifact (Block 32)

Conversion Notes

Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm is a dynamic INT4 variant (block-32 weights, FP32 activations). It was converted through the LiteRT Torch (litert-torch) path and quantized with AI Edge Quantizer. This artifact incorporates LiteRT-LM GPU graph optimizations, including composite ops for RoPE, fused QKV, and fused Gate/Up projections, and is configured with static prefill memory allocation.

Update (2026-08-31) — thought channel and think pre-fill repaired in place. Both bundles received two metadata-only repairs today (weights byte-identical each time, verified section by section). First, the thought channel (<think>\n / \n</think>) was declared in the bundle metadata — without it the runtime streams raw reasoning inline into the answer and silently ignores any thinking budget. Second, the generation prompt now pre-opens the think block exactly as the upstream Qwen3-4B-Thinking-2507 template does (<|im_start|>assistant\n<think>\n): model.litertlm's structured assistant prefix and the block-32 file's embedded template both previously ended at assistant\n, leaving the model to emit <think> on its own — a discipline quantized reasoning models lose first. After the repair model.litertlm scores 8/8 on the 8-question sanity gate on both macOS backends (CPU and Metal GPU, max-tokens 2048), and the block-32 file scores 8/8 on CPU.

⚠️ It's a reasoning model — give it room to think

This model generates a <think>…</think> reasoning chain, then the answer. Run it with max_tokens ≥ 2048 — at a short limit it gets cut off mid-thought and never reaches the answer. (All quality numbers below were measured at 2048.)

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The iPhone row is carried over from this repository's own earlier on-device note (LiteRT-LM Swift runtime); its run log is not retained here, so the run count and prompt are not known.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 2048, identical prompt and answer-extraction for every row).

int4 is at parity (−4 pt). Note: evaluating a reasoning model at a short token budget badly understates int4 — the longer int4 reasoning chains get truncated before the answer; benchmark reasoning models with max_tokens ≥ 2048.

Why block 128 (and not block 32)? For this reasoning model the block-32 build degraded more (−9 pt) and produced corrupted output under the iPhone GPU delegate, while block 128 is robust on every backend, ~40 % faster to decode (¼ the dequant scales — which matters when generating long <think> chains), and stays at −4 pt parity. So block 128 is the build we recommend; the block-32 file is also published (see Artifact (Block 32) above), and both bundles run fully delegated on the Galaxy S26 GPU.

The block-32 file's GPU corruption is not iPhone-specific: on macOS Metal it degenerates too (question-echo loops, truncated reasoning; measured 2026-08-31 on the file both before and after the metadata repair, so it is pre-existing and unrelated to the repair). Use the block-32 file on CPU; on Android GPU it delegated and generated in the Galaxy S26 gate above.

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. In model.litertlm, XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A bat and a ball cost \$1.10. The bat costs \$1.00 more than the ball. How much is the ball?"

The .litertlm bundle carries the tokenizer and prompt template (Qwen3 ChatML — <|im_start|>role\n…<|im_end|>, stop token <|im_end|>), so no separate tokenizer files are needed. The model produces a <think>…</think> block followed by its answer.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs .litertlm models on-device:

1. Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
2. Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/
3. In the app tap +, pick the file, choose the GPU backend, and raise the max-tokens setting (≥2048).
4. Chat — the bundle already carries the tokenizer and Qwen3 chat template.

A 4B int4 build needs ~2.5 GB free RAM; reboot the phone first if memory is tight.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Qwen3-4B-Thinking-2507 model.litertlm qwen3-4b-thinking-2507
litert-lm run qwen3-4b-thinking-2507     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone

Verified on iPhone 17 Pro (LiteRT-LM Swift runtime): loads and generates at ~14 tok/s.

Conversion

Converted with the official litert-torch converter — a standard Qwen3ForCausalLM, so it uses the existing Qwen3 path with no custom graph code. Recipe: blockwise-128 int4 + OCTAV (INT4 weights, block 128, symmetric, OCTAV optimal-clipping), embeddings INT8, KV cache 4096.

from litert_torch.generative.export_hf.export import export
export(
    model="Qwen/Qwen3-4B-Thinking-2507",
    output_dir="out",
    quantization_recipe="qwen3_int4_block128_octav.json",  # blockwise-128 int4 + OCTAV, int8 embeddings
    cache_length=4096,
    externalize_embedder=True,
)

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License

Apache-2.0, inherited from the base model Qwen/Qwen3-4B-Thinking-2507.