Qwen3-4B — LiteRT-LM bundles exported with litert-torch's GPU composite flags

Test exports made for the LiteRT team's GPU-composite work (2026-09-10; re-exported 2026-09-11 with the team's fixed-prefill recipe, see Export). Not a release: on LiteRT-LM 0.17.0's macOS GPU paths the full-flag file does not produce correct output yet (see Status below). Weights int4 blockwise-32, activations fp32 (dynamic_wi4b32_afp32).

Files

2026-09-21: chat template updated to accept the 0.18 content-parts form. It is the v0.17.0 pbtext's template plus a format_content macro (dual/4155a0aa5616.jinja), swapped in after the export below with swap_template.py. The prompt rendered from string content, the weights, the tokenizer and the executor metadata are byte-identical; the sha256 column is the swapped files'. The pre-swap files are this repo's previous commit.

Signatures: prefill_1024 (fixed) and decode; cache 32771 (from --cache_length=32768 with --enable_gpu_dynamic_cache; the exporter rounds GPU-dynamic sizes up to the next prime). Bundle metadata from LiteRT-LM's models/qwen3/LlmMetadataProto.pbtext (max_num_tokens 4096, sampler TOP_P k=20 p=0.95 t=0.6, stop <|im_end|>). The 2026-09-10 versions of these files (prefill_1031 from --enable_gpu_dynamic_prefill, exporter-generated metadata with max_num_tokens 32771) are in this repo's history at commit 301fd8d1748ffbbd9f492cff86eb30defe0bd47c. Composites in the 11-flag graph: odml.qkv_norm_rope, odml.swiglu, odml.cache_update, odml.sdpa_transposed (decode), with fused QKV / gate-up projections and a bool mask.

Export

litert-torch upstream/main 6d4c622 (2026-09-08; use_swiglu_composite is not in the 0.9.4 wheel), litert-lm-builder 0.17.0 (main's builder needs it), ai-edge-litert 2.2.0, ai-edge-quantizer 0.9.0, torch 2.13.0, transformers 5.14.1.

python -m litert_torch.generative.export_hf \
  --model=Qwen/Qwen3-4B --output_dir=out/p1024 \
  --quantization_recipe=dynamic_wi4b32_afp32 \
  --litert_lm_llm_metadata_override=<LiteRT-LM checkout>/models/qwen3/LlmMetadataProto.pbtext \
  --enable_gpu_dynamic_cache=True \
  --apply_gpu_composites=True --fuse_gate_up=True --fuse_qkv=True \
  --use_qkv_norm_rope_composite=True --use_swiglu_composite=True \
  --use_sdpa_composite=True --use_bool_mask=True \
  --bundle_litert_lm=True --prefill_lengths=1024 --cache_length=32768

The 9-flag file drops the two --use_*_composite flags named in the table.

Status on LiteRT-LM 0.17.0 (macOS, Mac Studio M4 Max)

The speeds below were measured 2026-09-10/11 on the previous versions of these files (dynamic prefill prefill_1031, exporter-generated metadata; graphs otherwise identical). The current fixed-1024 files were re-checked 2026-09-11: 9-flag files 8/8 on the 8-question check, 11-flag failure modes unchanged, and the 0.6B A/B against its previous 9-flag file (three interleaved passes, same host) read the same speed within 3 %, and the 4B file's graph differs from its previous version in the same way.

Android and iOS: not measured.

Run (macOS)

pip install litert-lm==0.17.0
litert-lm run Qwen3-4B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu --thinking false --cache no --prompt "What is the capital of Japan? Answer briefly."
litert-lm benchmark Qwen3-4B_dynamic_wi4b32_afp32_9flags.litertlm --backend gpu -p 1024 -d 128 --cache no

Known runtime issues

With litert-lm 0.19.0 or newer, create the Engine with max_num_tokens of 4096 for these files. They have only a 1,024-token prefill signature, and 0.19.0 stops the first prefill with FAILED_PRECONDITION when max_num_tokens is 1,024 (0.18.1 runs it).

The 11flags file needs litert-lm 0.19.0 or newer for the GPU: on 0.18.1 it returns garbage on the Mac GPU (WebGPU), while the 9flags file answers correctly on both 0.18.1 and 0.19.0 (one prompt, thinking off, max_num_tokens 4096).

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.