Sign up

litert-community/Qwen3.5-0.8B

🤗 On Hugging Facetext-generationapache-2.04.2 GBother✓ HF checksums availableupdated today
Magnet

Qwen3.5-0.8B — LiteRT-LM

Update 2026-08-13: re-converted from the same weights with the gated-delta-rule chunk kernel re-expressed in an all-rank-≤-4, pad-free form the GPU delegate executes correctly. The graph now delegates fully to the GPU and answers correctly there — verified end-to-end on macOS and iPhone 17 Pro (Metal), where GPU decode runs ~2.8× the CPU. CPU behaviour and speed are unchanged. GPU execution requires fp32 activations, which the bundle now declares; see the honest notes for the memory cost. (While rewriting we found the GPU delegate miscomputing one op shape — reported upstream as LiteRT#9272; the kernel now avoids that shape.)
Update 2026-08-07: metadata-only fix (weights unchanged) — <|im_end|> is now declared as a stop token alongside <|endoftext|>. Qwen3.5 uses different tokens for the chat turn-end and config.json's eos_token_id; with only the latter declared, the literal <|im_end|> text leaked into litert-lm run output (generation still stopped correctly). Replies are now clean, and multi-turn history no longer records the marker as text.

Qwen/Qwen3.5-0.8B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). To our knowledge this is the first Qwen3.5 in LiteRT form, and the first gated-delta-net hybrid served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its delta-rule scan on a mobile GPU.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 0.8B model has 18 linear-attention + 6 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 6 attention layers keep KV (4096-token budget here). The upstream 0.8B checkpoint is multimodal; this package is the text decoder only (the vision tower and MTP heads are dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

| File | Recipe | Size |

|---|---|---|

| Qwen3.5-0.8B_int8.litertlm | int8 dynamic on linears + embedding (convs and the delta rule stay float) | 963 MB |

| Qwen3.5-0.8B-VL_int8.litertlm | text + image — same decoder recipe, plus the checkpoint's own ViT (fp16 encoder, int8 adapter), static 512×512, six-signature prefill ladder | 1.30 GB |

Correctness

All gates below were run on this exact file (or its float parent) after the 2026-08-13 rewrite:

  • Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
  • 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). On iPhone 17 Pro the GPU run answers 8/8.
  • Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
  • Multi-turn conversations carry state correctly across turns.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for both rows (non-thinking mode, matching the bundled template):

| Configuration | GSM8K |

|---|---|

| PyTorch bf16 (reference, MPS) | 12% |

| LiteRT int8 (this file) | 11% |

The absolute level is the 0.8B non-thinking model's own (thinking disabled, 512-token budget); the one-point gap is what the conversion + int8 cost.

Usage

litert-lm run ./Qwen3.5-0.8B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-0.8B_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty \n\n block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

| Backend | Prefill (256) | Decode | TTFT |

|---|---|---|---|

| GPU | 1972 tok/s | 161.8 tok/s | 0.14 s |

| CPU | 666 tok/s | 46.7 tok/s | 0.41 s |

CPU figures are essentially unchanged from the previous file (the delta-rule kernel was already matmul-form on CPU); the 2026-08-13 update adds the working GPU path.

On device (cold start, single runs, 138-token prompt, quality-gate harness):

| Device | Backend | Prefill | Decode | TTFT | Peak memory |

|---|---|---|---|---|---|

| iPhone 17 Pro | GPU (Metal) | 387 tok/s | 41.4 tok/s | 0.47 s | 5.48 GB |

| iPhone 17 Pro | CPU | 170 tok/s | 14.6 tok/s | 0.93 s | 1.21 GB |

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 260-token prompt with generation running long, 3 runs):

| Backend | Prefill (260 tok) | Decode | TTFT |

|---|---|---|---|

| CPU | 50–134 tok/s | 8.0–13.6 tok/s | 2.0–5.3 s |

| GPU (OpenCL) | — does not fit this phone's memory (see honest notes) | | |

Honest notes:

  • Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). Qualcomm Adreno: verified once, on a Galaxy S26 (SM8850, litert-lm 0.16.0, 2026-08-24) — full delegation and a correct answer, 30.9 tok/s decode, but a 110 s engine init and a 5.4 GB peak, so on Snapdragon phones the CPU backend is still the practical choice. The vision file Qwen3.5-0.8B-VL_int8.litertlm was gated the same way on that phone on 2026-09-05: GPU 35 tok/s decode against 33 on the CPU, with a 74 s engine init and 2.1× the CPU's peak — CPU again. Same-device rows are in litertlm_manifest.json.
  • GPU inference runs with fp32 activations (declared in the bundle; an fp16-activation formulation is unfinished — the residual issue is a real-weight fp16 range overflow in one layer-0 head, a property of the checkpoint, not of the conversion). That is where the GPU memory multiple comes from — 5.48 GB vs 1.21 GB on iPhone above.
  • Pixel 8a cannot compile this file on its GPU: fp32-expanded weights plus the full prefill-ladder of compiled programs exceed the phone's ~3.8 GB available memory (a reduced dev build of the same graph runs and answers correctly there, fully delegated — the limit is memory, not ops). CPU works as before on Android.
  • On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
  • On quality: the per-question sanity gate is word-for-word identical to HF fp32, but on a harder composite probe (8 questions in one 138-token prompt) the int8 quantization measurably costs answers at this 0.8B scale. If you need maximum fidelity on complex prompts, ask for a float/fp16 variant.

Vision build (Qwen3.5-0.8B-VL_int8.litertlm)

The upstream checkpoint ships a 12-layer, 768-dim ViT that the text-only package drops. This build keeps it, wired to LiteRT-LM's fast_vlm contract: one 512×512 image per turn, 1024 patches merged 2×2 into 256 soft tokens injected at the image position. There is no DeepStack in this tower (deepstack_visual_indexes is empty upstream), so the single image embedding is the whole visual signal.

litert-lm run ./Qwen3.5-0.8B-VL_int8.litertlm \
  --prompt "What is in this image?" --attachment ./photo.png

# vision on the GPU as well as the decoder
litert-lm run ./Qwen3.5-0.8B-VL_int8.litertlm --backend gpu --vision-backend gpu \
  --prompt "Where is this scene?" --attachment ./photo.png --cache no

Text-only prompts work on this file too; it is a superset of the text build, at a larger download and a larger GPU footprint.

Measured

Mac, litert-lm benchmark (0.16.0), M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine — text path:

| Backend | Prefill (256) | Decode | TTFT |

|---|---|---|---|

| GPU | 1890 tok/s | 126.2 tok/s | 0.15 s |

| CPU | 649 tok/s | 46.6 tok/s | 0.44 s |

iPhone 17 Pro, cold start, single runs, quality-gate harness — the vision rows are real image turns:

| Backend | Task | Decode | TTFT | Peak memory | Load |

|---|---|---|---|---|---|

| GPU (Metal) | image + question | 65.8 tok/s | 1.5 s | 3.63 GB | 29.9 s |

| GPU (Metal) | image + question | 65.2 tok/s | 1.4 s | 3.84 GB | 30.4 s |

| GPU (Metal) | text (8-question probe) | 45.7 tok/s | 1.7 s | 3.77 GB | 30.5 s |

| CPU | text (8-question probe) | 26.0 tok/s | 2.0 s | 0.63 GB | 8.5 s |

Grounding on device was checked with two probes whose answers are opposite: an image containing text ("does this image contain text?" → "Yes, this image…") and one without ("No, this image…"). Both correct on Metal, with the vision encoder on the GPU.

Honest notes

  • The six-signature ladder fit Metal on the first try. The 2B sibling had to be re-exported at 1024, 256, 64, 16, 4, 1 after all three of its Metal legs were killed by jetsam; this build ships the same reduced ladder and needed no maxNumTokens override, no retry, and no second export. At roughly 40% of the 2B's file size it peaks at ~3.8 GB against the 2B's ~4.6 GB.
  • Positions are 1-D. The fast_vlm contract feeds sequential positions, so the checkpoint's M-RoPE collapses to plain RoPE. Expect fluent same-content paraphrase rather than token-exact agreement with a full M-RoPE reference — counting and dense-layout questions are where it shows.
  • The vision graph is faithful, and this tower quantizes better than the 2B's. The exported tower matches the model's own vision path at correlation 0.9999999999 in fp32 (max abs diff 1.05e-4). Quantizing the encoder to int8 holds 0.9916–0.9951 on real photographs — noticeably better than the 2B tower's 0.974–0.984 at the same recipe. This build still ships the encoder in fp16 and only the adapter in int8, matching the family, but the int8-encoder variant is on firmer ground here if a Mali device needs it.
  • The fp16-safe LayerNorm profile is this checkpoint's own, not the 2B's. Scales reach 16 from block 6 on (with 512 at the final norm), where the 2B reaches 32 from block 11. The scales are calibrated per export; copying the sibling's table would silently mis-scale the tower.
  • Quality on the composite 8-question probe, separated by cause — and the split runs the other way from the 2B. On Mac this file answers 8 of 8 on GPU and 8 of 8 on CPU, with the questions asked one at a time and as one combined prompt. On iPhone it scores 6/8 on Metal and 3/8 on CPU. Not one of the device misses reproduces on Mac, so they are device-side rather than the conversion — the opposite of the 2B, whose arithmetic slip did reproduce on Mac and was therefore the model. The mechanism behind the device degradation is not identified, and this card does not guess at one. On iPhone, prefer the GPU backend.
  • The iPhone CPU 3/8 is reproducible, not thermal. It was first measured on a warm handset; re-run three days later from cold, after an app reinstall, the answers are byte-identical. Peak memory is the figure that moved (0.63 GB cold against 0.91 GB warm).
  • Android is not gated for this build. On Mali the fp16 vision encoder is known to crash the device on other models of this shape — an int8-vision build would be the Android path, and we have not measured one on a phone.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

  • Rank-4 chunk kernel (the 2026-08-13 change): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op — the GPU delegate miscomputes rank-3 non-final-axis PAD (reported as LiteRT#9272). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU.
  • Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
  • State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/recurrent state consumed, so multi-chunk prefill composes) and the decode graph traces the fused single-step branch (conv window rolled in place by causal_conv1d_update).
  • Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the delta rule (per-token decay forced to ~1, zeroed k/v injection) and the stored conv window is gathered at the last valid column. GPU trap: reductions in the guard keep their batch dimension (keepdim=True) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
  • Constant-eye chunk kernel: the reference kernel builds torch.eye inside the traced function, which lowers to a STABLEHLO_IOTA op no released TFLite kernel set registers; the identity matrix is lifted as a graph constant.
  • Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
  • Quantization: post-hoc dynamic int8 over linears + embedding only; the convs and the delta rule stay float.
  • GPU activation precision: the bundle's TOML declares prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |

|---|---:|---:|---:|---:|

| Qwen3.5-0.8B-VL_int8.litertlm | 71.6 (71.1–71.9) | 6.9 (6.9–7.0) | 3.8 s | 2.1 GB |

| Qwen3.5-0.8B_int8.litertlm | 70.5 (70.3–70.6) | 7.1 (6.8–7.1) | 3.8 s | 2.5 GB |

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: text-decoder weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; vision tower and MTP weights omitted; tokenizer repackaged unmodified; chat template replaced with the simplified ChatML template described above. This repository is a community conversion and is not affiliated with Alibaba / the Qwen team.