Sign up

litert-community/Qwen3-4B-Thinking-2507

🤗 On Hugging Facetext-generationapache-2.017 GBother✓ HF checksums availableupdated today
Magnet

Qwen3-4B-Thinking-2507 — LiteRT-LM (blockwise int4)

Qwen/Qwen3-4B-Thinking-2507 converted to

the LiteRT-LM (.litertlm) format for on-device inference with Google's

LiteRT-LM runtime (the engine behind the official

litert-community/* models).

Qwen3-4B-Thinking-2507 is a dense 4B reasoning model (Qwen3ForCausalLM, 36 layers) that

operates exclusively in thinking mode — it emits a … chain before its answer —

so it rides the existing Qwen3 converter and runtime directly.

Artifact (Block 128)

| | |

|---|---|

| File | model.litertlm — int4 block 128 (~2.3 GB) |

| Quantization | int4 weights (symmetric) + OCTAV optimal-clipping; embeddings INT8 (externalized section) |

| Compute | integer |

| Context (KV cache) | 4096 |

| Base model | Qwen/Qwen3-4B-Thinking-2507 |

Artifact (Block 32)

| File | Quantization Recipe | Context | Size |

|---|---|---:|---:|

| Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm | dynamic_wi4b32_afp32 (block-32) | 4096 | 2.1 GB |

Conversion Notes

Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm is a dynamic INT4 variant (block-32 weights, FP32 activations). It was converted through the LiteRT Torch (litert-torch) path and quantized with AI Edge Quantizer. This artifact incorporates LiteRT-LM GPU graph optimizations, including composite ops for RoPE, fused QKV, and fused Gate/Up projections, and is configured with static prefill memory allocation.

Update (2026-08-31) — thought channel and think pre-fill repaired in place. Both bundles received two metadata-only repairs today (weights byte-identical each time, verified section by section). First, the thought channel (\n / \n) was declared in the bundle metadata — without it the runtime streams raw reasoning inline into the answer and silently ignores any thinking budget. Second, the generation prompt now pre-opens the think block exactly as the upstream Qwen3-4B-Thinking-2507 template does (<|im_start|>assistant\n\n): model.litertlm's structured assistant prefix and the block-32 file's embedded template both previously ended at assistant\n, leaving the model to emit ` on its own — a discipline quantized reasoning models lose first. After the repair model.litertlm` scores 8/8 on the 8-question sanity gate on both macOS backends (CPU and Metal GPU, max-tokens 2048), and the block-32 file scores 8/8 on CPU.

⚠️ It's a reasoning model — give it room to think

This model generates a … reasoning chain, then the answer. **Run it with

max_tokens ≥ 2048** — at a short limit it gets cut off mid-thought and never reaches the answer.

(All quality numbers below were measured at 2048.)

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

| Device | Backend | Prefill (256) | Decode | TTFT | Load | Peak footprint |

|---|---|---|---|---|---|---|

| Apple M4 Max (macOS) | CPU | 111 tok/s | 17.8 tok/s | 2.48 s | — | — |

| Apple M4 Max (macOS) | GPU (Metal) | 999 tok/s | 68.5 tok/s | 0.28 s | — | — |

| iPhone 17 Pro | GPU (Metal) | — | ~14 tok/s | — | — | — |

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The iPhone row is carried over from this repository's own earlier on-device note (LiteRT-LM Swift runtime); its run log is not retained here, so the run count and prompt are not known.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 2048, identical prompt

and answer-extraction for every row).

| Configuration | GSM8K |

|---|---|

| bf16 (reference) | 90.0% |

| LiteRT int4 — block 128 | 86.0% (−4 pt) |

int4 is at parity (−4 pt). Note: evaluating a reasoning model at a short token budget badly

understates int4 — the longer int4 reasoning chains get truncated before the answer; benchmark

reasoning models with max_tokens ≥ 2048.

Why block 128 (and not block 32)? For this reasoning model the block-32 build degraded more

(−9 pt) and produced corrupted output under the iPhone GPU delegate, while **block 128 is robust

on every backend, ~40 % faster to decode** (¼ the dequant scales — which matters when generating

long `` chains), and stays at −4 pt parity. So block 128 is the build we recommend; the block-32 file is also published (see Artifact (Block 32) above), and both bundles run fully delegated on the Galaxy S26 GPU.

The block-32 file's GPU corruption is not iPhone-specific: on macOS Metal it degenerates too (question-echo loops, truncated reasoning; measured 2026-08-31 on the file both before and after the metadata repair, so it is pre-existing and unrelated to the repair). Use the block-32 file on CPU; on Android GPU it delegated and generated in the Galaxy S26 gate above.

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

| file | GPU backend | delegation | peak |

|---|---|---|---:|

| Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm | runs | 3764 / 3764 ops across 3 subgraphs on LiteRT GPU | 1627 MB |

| model.litertlm | runs | 3270 / 3270 ops across 2 subgraphs on LiteRT GPU | 971 MB |

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. In model.litertlm, XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A bat and a ball cost \$1.10. The bat costs \$1.00 more than the ball. How much is the ball?"

The .litertlm bundle carries the tokenizer and prompt template (Qwen3 ChatML —

<|im_start|>role\n…<|im_end|>, stop token <|im_end|>), so no separate tokenizer files are

needed. The model produces a … block followed by its answer.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs

.litertlm models on-device:

1. Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).

2. Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/

3. In the app tap +, pick the file, choose the GPU backend, and raise the max-tokens setting (≥2048).

4. Chat — the bundle already carries the tokenizer and Qwen3 chat template.

A 4B int4 build needs ~2.5 GB free RAM; reboot the phone first if memory is tight.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official

LiteRT-LM CLI — including as a

local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Qwen3-4B-Thinking-2507 model.litertlm qwen3-4b-thinking-2507
litert-lm run qwen3-4b-thinking-2507     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone

Verified on iPhone 17 Pro (LiteRT-LM Swift runtime): loads and generates at ~14 tok/s.

Conversion

Converted with the official litert-torch

converter — a standard Qwen3ForCausalLM, so it uses the existing Qwen3 path with no custom graph

code. Recipe: blockwise-128 int4 + OCTAV (INT4 weights, block 128, symmetric, OCTAV

optimal-clipping), embeddings INT8, KV cache 4096.

from litert_torch.generative.export_hf.export import export
export(
    model="Qwen/Qwen3-4B-Thinking-2507",
    output_dir="out",
    quantization_recipe="qwen3_int4_block128_octav.json",  # blockwise-128 int4 + OCTAV, int8 embeddings
    cache_length=4096,
    externalize_embedder=True,
)

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |

|---|---:|---:|---:|---:|

| Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm | 8.9 (8.8–9.0) | 1.4 (1.4–1.4) | 29.4 s | 4.4 GB |

| model.litertlm | 10.6 (10.5–10.7) | 1.5 (1.5–1.5) | 25.5 s | 3.9 GB |

License

Apache-2.0, inherited from the base model

Qwen/Qwen3-4B-Thinking-2507.