Qwen3.8-27B — LiteRT-LM

Which file: Qwen3.8-27B_mixed_int4.litertlm is the file for the LiteRT-LM releases (0.18.1 to 0.19.1) and for the CPU; it also runs on LiteRT-LM main. Qwen3.8-27B_mixed_int4_gpu_composite_channels_last.litertlm (added 2026-11-07) is a GPU file whose odml.short_conv_step composites keep their conv state channels-last, the layout LiteRT's GPU path takes from LiteRT #11565 on. No LiteRT-LM release includes #11565 yet: today the file runs on LiteRT-LM main (checked at b1dcfbd0) and with the litert-lm-nightly 0.20.0.dev20261106 package, and LiteRT-LM 0.19.1 stops with an error when it creates the GPU engine for it. It becomes the default file here once a release includes #11565.

This repository is a community conversion of Qwen/Qwen3.8-27B to the LiteRT-LM (.litertlm) format, made with litert-torch (commit 115a1360 with the Qwen3.5 hybrid patch of hf-to-litertlm) for Google's LiteRT-LM runtime 0.18.1. On a Mac Studio M4 Max the Mixed INT4 file prefills 1,024 tokens at 175.5 tok/s and decodes at 18.2 tok/s on the GPU (measured 2026-10-22). It is one 15.4 GB file holding the text decoder, for desktop-class machines: it was measured on a Mac only.

Qwen introduces Qwen3.8 as "the most capable generation in the Qwen open-model family to date", built "on the architectural foundation of Qwen3.5", and Qwen3.8-27B as "a compact, deployment-friendly dense model: a native vision-language model that understands images and videos, with flexible thinking control". Qwen's card reports GPQA Diamond 89.2 and IFBench 79.5 for it (vendor-reported, not run here) and recommends sampling with temperature 1.0, top_p 0.95 and top_k 20 in the thinking mode and temperature 0.7, top_p 0.8, top_k 20 and presence_penalty 1.5 in the non-thinking mode.

The architecture is Qwen3.5's hybrid at 27B: 64 layers in 16 groups of three GatedDeltaNet linear-attention layers and one gated full-attention layer, each with a dense FFN; hidden size 5120, FFN size 17,408, 24 attention heads with 4 KV heads of 256 (rotary position on 64 of the 256 dimensions), linear attention with 16 query/key heads and 48 value heads of 128, a 248,320-token vocabulary with untied embeddings. The upstream checkpoint is multimodal and has a multi-token-prediction head; this file holds the text decoder only (26,895,998,464 of the release's 27,781,427,952 parameters, the same text-only load as transformers' Qwen3_5ForCausalLM), so the image and video input of Qwen3.8-27B and its multi-token prediction are not available here. Only the 16 attention layers keep a growing KV cache (a 4,096-token budget in this file); the 48 linear-attention layers carry a constant-size conv and recurrent state.

Qwen3.8 thinks by default. This file runs with thinking off only: its chat template writes Qwen's chat format with an empty <think>\n\n</think>\n\n block at every generation prompt (it does not read enable_thinking or reasoning_effort), the bundle declares no thought channel, and supports_thinking is false. Thinking on is left out because it broke multi-turn conversations at litert-lm 0.18.1's default settings: with a thinking switch in the same file (thinking on, thinking_token_budget=300, Mac GPU), the third turn of "My name is Kenji." / "What is 17 times 23?" / "What is my name?" came back empty in 3 of 3 runs, while with the conversation setting filter_channel_content_from_kv_cache turned off (it is on by default) the same file answered "Your name is Kenji." (one run, 2026-10-23), as the bf16 model in transformers did.

2026-11-03: this file's LLM metadata now carries the tool-call format of LiteRT-LM v0.19.0's models/qwen3_5 configuration (tool_call_config, supports_function_calling: true), so LiteRT-LM 0.19.0 returns the model's tool calls as tool calls. Nothing else in the LLM metadata changed (the thinking-off template, stop tokens, model type), and the TFLite, executor-metadata and tokenizer sections are byte-identical. sha256: 7396292e… → 5273a07d… (size unchanged). Checked with litert-lm 0.19.0 on a Mac CPU (Python API, greedy, a fresh process per prompt): 3 of 3 tool prompts come back as tool calls (the previous file returned the call as text), 3 plain replies are byte-identical to the previous file's, thinking on and off behave as before (no reasoning either way), and litert-lm 0.18.1 still loads the file. Every other result on this card was measured on earlier files.

2026-11-07: added Qwen3.8-27B_mixed_int4_gpu_composite_channels_last.litertlm, a new export of the same checkpoint with litert-torch main at d600176b (with three changes that cut the export's memory and leave the graph as main writes it), whose Qwen3.5 graph has GPU composites, odml.gated_delta_block and a channels-last odml.short_conv_step among them. Same signatures and KV budget as this file; its LLM metadata (thinking off, the 2026-11-03 tool-call format) and its tokenizer section are byte for byte this file's. On the Mac M4 Max GPU it answered the 8 questions under Correctness 8/8 with no degenerate answer, one process each, with LiteRT-LM main b1dcfbd0 built for macOS and with litert-lm-nightly 0.20.0.dev20261106; litert-lm 0.19.1 refuses it when it creates the engine (Following operations are not supported by GPU delegate, then Failed to create engine). Not run on the CPU, a phone or iOS. Everywhere else on this card, "this file" is Qwen3.8-27B_mixed_int4.litertlm; on 2026-11-07 it also loaded on the Mac GPU with litert-lm 0.19.1 and answered the 8 questions 8/8 with a Mac build of LiteRT-LM main 1d378fca with LiteRT bf49eb62, the LiteRT that main pins from b1dcfbd0 on.

Time for a 1024-token prefill plus 256 decoded tokens at context 2048, computed from the rates in the Performance table below: 19.87 s on the Mac GPU, 92.74 s on the Mac CPU (with the CPU weight cache, see Usage). On GSM8K (thinking off, 2,048 output tokens, table in Accuracy) the file scored 97 on the Mac GPU against 96 for the bf16 model in transformers.

Correctness

Unless a line names the file published on 2026-10-23, the checks and the benchmarks on this card ran on an earlier build of this file that still carried a thinking switch in its metadata; that file differs from it only in the LlmMetadata section (the template, the channel and supports_thinking), with the TFLite, tokenizer and executor sections byte for byte the same, and the published template renders the same thinking-off prompts as the earlier one, byte for byte, in nine conversation shapes (216 of 216 renders). On that file itself, on the Mac GPU: the 8 questions below 8/8 with the same answers, and the three-turn conversation with thinking off ("Give me one word that means 'very big'." / "Use it in a short sentence." / "Now make that sentence a question.") answered "Enormous" / "The elephant was enormous." / "Was the elephant enormous?" with no chat marker in the text.

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each, thinking unset): 8/8 on the CPU and 8/8 on the GPU, no degenerate answer (none empty, none a run of one repeated word or character) (42 / Tokyo / Cold / 7 / Merci / 56 / 0.9 / blue). With the runtime's default fp16 GPU activations instead of the declared fp32 ones, the same weights answered the first question with a run of ! to the token cap (one run, 2026-10-22; the check stops at its first failure), which is why the file declares fp32 activations.

Decode walk against the bf16 model (transformers 5.14.1 Qwen3_5ForCausalLM, bf16 on the CPU; teacher-forced over the first 24 positions of a GSM8K question in the chat format, both models fed the same prompt tokens one at a time; this file's TFLite section through LiteRT's CompiledModel API on the CPU): the top token is the same at 23 of 24 positions and within the top 5 at all 24; the one miss is a near tie in the reference (its top two logits 0.25 apart). Mean KL divergence 0.0356, mean Pearson correlation of the logits 0.973 (lowest 0.935), largest logit difference 4.81.

A multi-digit multiplication check on the CPU (six questions, three fresh processes each, thinking off, greedy, 400-token cap) ran on the block-32 int4 file of the same export, which is not published here: 18 of 18 right, every run ending by itself with no repeated line. On this file the CPU checks are the 8 questions above and the decode walk.

The three-turn conversation with thinking off (Correctness, first paragraph) also passed on the earlier file: on the Mac GPU with the same three replies, which are the bf16 model's in transformers word for word, and on the Mac CPU (with the weight cache) with "Huge" / "The house was huge." / "Was the house huge?".

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's template (thinking off), greedy, at most 2,048 output tokens, a 4096-token KV budget, a fresh process per question with at most 3 at once (litert-lm 0.18.1 Python API, Mac GPU, on the earlier file with the same thinking-off prompts). The bf16 reference is transformers 5.14.1 on MPS with Qwen's template at enable_thinking=False, the same prompts, greedy, 2,048 new tokens. The answer is the number after the last ####, else the reader script's next rules (\boxed{}, "answer is", the last number). IFBench and GPQA were not run; Qwen's own table reports them above.

Question by question the two give the same answer on 98 of 100. Question 12 is counted right only in this file, and its reply is not a clean one: it reached the 2,048-token cap without a final line after writing "answer is 13" and "answer is 12" twice each, and the reader script read the last of these, 13. Question 62 is wrong in both, with different answers. 15 of this file's replies ended without the final line and were read by the next rules, all of them counted right; question 12 is the only one of its 100 replies that reached the cap, and none of the 100 processes logged a WebGPU validation error. Three bf16 replies (45, 62 and 93) reached the cap.

Usage

litert-lm run ./Qwen3.8-27B_mixed_int4.litertlm --prompt "What is 17 + 25? Answer briefly." --backend gpu --cache no
litert-lm run ./Qwen3.8-27B_mixed_int4.litertlm --prompt "What is 17 + 25? Answer briefly." --backend cpu --cache disk
litert-lm benchmark ./Qwen3.8-27B_mixed_int4.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend gpu

Run as printed on the Mac M4 Max with the file under this name (litert-lm 0.18.1, 2026-10-23), the two run commands printed 42 and exited 0, and the benchmark exited 0 with prefill 176.99 and decode 18.42 tok/s.

• The GPU engine takes 3½ to 4 minutes to build on the Mac M4 Max: the first command above took 217 s from start to answer, and a benchmark process spends 214–247 s per engine (three processes, from their wall times); the CLI prints about twice that as its init time because it adds up overlapping init phases (LiteRT-LM issue 4057). Build it once and keep it for the conversation.
• The CPU needs the weight cache. Without it (--cache no) the CPU engine packs the int4 weights once for each of the 7 signatures: on the block-32 file of the same export the benchmark's memory footprint grew from 5.8 GB to 91 GB in 48 s during the engine build on the 128 GiB Mac, with swap growing, and the run was stopped (one run, 2026-10-22). A CPU decode walk of the block-32 file's TFLite model through LiteRT's CompiledModel API (2026-10-21) peaked at 101.8 GB without the cache and at 5.2 GB with it. With --cache disk the first CPU run writes a 14.0 GB XNNPACK weight cache next to the model (14,047,805,808 bytes) and later runs read it: the second command above, run in a directory without the cache, took 120 s at a 4.2 GB peak memory footprint while it wrote the cache; with the cache in place the benchmark's peak was 4.1 GB and its engine build 122–125 s.
• One Engine per conversation. LiteRT-LM issue 3165 (open): a new Conversation on an Engine that already ran another conversation can continue from the earlier conversation's recurrent state. The probe on this file (a pirate-persona conversation, then one told to answer in one word, on the same Engine and on a fresh Engine; litert-lm 0.18.1, CPU, greedy) kept the one-word rule 3 of 3 times in each of 3 shared-Engine runs and in the fresh-Engine run, so the effect did not show here; create a new Engine for each conversation all the same. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05, the internal CL that continued PR #3945). No release carries it yet (v0.19.1 does not); this note will be updated when one does.
• One Engine per process on the Mac GPU. The benchmark builds 4 engines one after another in one process. Across 11 such processes on this file and on variants of it, WebGPU validation errors appeared in 2 of the 44 engines, each time in the third engine of its process (on the block-32 file and on a copy without the fp32 declaration). No error appeared in this file's 12 benchmark engines, in 115 single-engine processes on this file (the 100 GSM8K questions among them) or in 15 single-engine processes on the variants.
• Without --backend, run uses the bundle's default backend, the CPU. The bundle sets no sampler and the checks on this card ran greedy; to sample as Qwen recommends for the non-thinking mode, pass --top-k, --top-p and --temperature together (--top-k 20 --top-p 0.8 --temperature 0.7).
• Tool calling: the template renders a tool list in Qwen's format and a tool message as a <tool_response>. Since 2026-11-03 the file declares supports_function_calling: true and the tool-call configuration of LiteRT-LM v0.19.0's models/qwen3_5 (the regex configuration main added on 2026-10-20, the fix for issue 3927): LiteRT-LM 0.19.0 returns a call the model writes as a tool call, with the name and the arguments (Python API, Mac CPU; the 2026-11-03 paragraph above). litert-lm 0.18.1 and earlier have no field for that configuration (read in the LiteRT-LM source): there the bundle's model type (generic_model) has no parser for Qwen's XML tool call, a call comes back as text, and an app that needs the calls parses that text itself, although litert-lm 0.18.1 reports supports_function_calling as true. With tools in the conversation, LiteRT-LM 0.19.0 returns an error (INVALID_ARGUMENT, "Failed to parse tool calls from code block") instead of the reply when the model writes a closed <tool_call>…</tool_call> block without a <function=…> call; in 0.19.0 only the C++ API can turn this off (SetReturnErrorOnParseFailure). Seen on 2026-11-03 (Mac CPU, Python API) with Qwen3.5-4B's Mixed INT4 file, which carries the same tool-call configuration.
• Replies that take over 10 minutes. On litert-lm 0.18.x and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower max_output_tokens or use the streaming send_message_async, which has no such limit.

For Qwen3.8-27B_mixed_int4_gpu_composite_channels_last.litertlm, until a release includes LiteRT #11565:

pip install litert-lm-nightly==0.20.0.dev20261106
litert-lm run ./Qwen3.8-27B_mixed_int4_gpu_composite_channels_last.litertlm --prompt "What is 17 + 25? Answer briefly." --backend gpu --cache no

Run as printed on the Mac M4 Max in a new Python 3.12 virtual environment (2026-11-07), the run command printed 42 and exited 0, 27 s from start to answer (the first run in that environment).

Mac app: litertlm-chat-mac is a SwiftUI chat app that opens this file and answers on the Mac's GPU through LiteRT-LM's Swift package (0.18.1), one engine process per conversation. In the app on the Mac M4 Max (2026-10-23 and 24), the file decoded at 18.29–18.53 tok/s on answers of 301–942 tokens and gave the Correctness section's answers to its 8 questions and its three-turn conversation.

Performance

Mac Studio M4 Max (128 GiB), litert-lm 0.18.1: the benchmark command above with --backend gpu or cpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations; two benchmark processes per backend, each in a measurement window of its own (the other jobs on the Mac held back), the mean of the two. GPU is WebGPU on Metal with --cache no; the CPU row is --cache disk with the weight cache already written (Usage). Engine build = the process's wall time minus 4 × the time of one 1024 + 256 workload (from the printed rates), divided by the 4 engines a benchmark process builds (one per run, the warmup included); peak = the process's peak memory footprint.

The two GPU processes measured 19.75 and 19.99 s for the workload, the two CPU processes 92.96 and 92.51 s. The block size was chosen on this Mac: the block-32 file of the same export (16.52 GB) took 21.31 s on the GPU and 103.34 s on the CPU, and both files answered the 8 questions 8/8 on both backends and matched the bf16 model's top token at 23 of 24 walk positions. The runtime's default fp16 GPU activations ran the workload in 17.32 s (one benchmark process) but broke the answers (Correctness). A three-signature variant (prefill 1024, 256 and 1) built its GPU engine in 134–139 s at the same speed (20.00 s) and is not published: LiteRT-LM's prefill planner hands a remainder under half of a prefill size to the next smaller one, so with those sizes a remainder under 128 tokens would run one token per call.

The row of the 2026-11-07 file used the same method with that file's name and the litert-lm-nightly 0.20.0.dev20261106 CLI (2026-11-07): its two processes measured 15.11 and 15.12 s for the workload and built their engines in 6.0 and 1.5 s. Qwen3.8-27B_mixed_int4.litertlm was not benchmarked on that runtime, so the two GPU rows differ in the runtime as well as the file. Speed is still being tuned (updates follow).

• Not run on a phone or on iOS: the file alone (15.38 GB) does not fit in the memory of the phones we test (MemTotal 11,389,756 kB on a Galaxy S26, 7,754,700 kB on a Pixel 8a); it was measured on a desktop only.
• Not run on an NPU: LiteRT has no NPU accelerator on macOS.

Conversion notes

• Source and stack: Qwen/Qwen3.8-27B at revision 1d4bf0f2 (every file of the revision checked against the Hub's record); litert-torch at commit 115a1360 with the Qwen3.5 hybrid patch of hf-to-litertlm, transformers 5.14.1, torch 2.12.1, ai-edge-quantizer 0.8.0; packed and checked with litert-lm 0.18.1. qwen35_work/convert_qwen38_27b.sh in hf-to-litertlm is the conversion as one command; run from the download to the checks on 2026-10-22 and 10-23 for Qwen/Qwen3.5-2B, a smaller checkpoint of the same architecture, it finished with exit 0 and its file's metadata section came out byte for byte as the LlmMetadata this file had before 2026-11-03; qwen35_work/use_tool_call_config.sh is the 2026-11-03 step that adds the tool-call format and packs with the litert-lm 0.19.0 CLI.
• Export: the text decoder through litert-torch's lightweight conversion (constants of 1e8 bytes and more stay lazy), 6 prefill signatures (1024, 256, 64, 16, 4, 1) plus decode over a 4096-position KV budget, 128 state buffers (16 key and 16 value caches, 96 linear-attention states). transformers loads a bf16 checkpoint for a float32 export by making a float32 copy of every weight in RAM (about 108 GB here). The export read an exact F32 copy of the checkpoint through a memory map instead. litert-torch's in-place clamp of every float constant would have written into that map, so it was replaced by a read-only ±inf check (no constant held ±inf, so the constants are the same). The export took 66 minutes with an 11.8 GB peak memory footprint on the 128 GiB Mac and wrote only the 107.7 GB float TFLite, no float .litertlm.
• Quantization after the export: int4 blockwise-128 min-max on the 400 linear layers of the 64 blocks, the lm_head and the embedding int8 channelwise, the 96 decay/gate projections of the 48 linear-attention layers kept float (as in the smaller Qwen3.5 files, where these small heads did not take the CPU backend's run-time int8 quantization of their input). 694 of the 190,054,400 block scales were raised to a floor of 2^-14; every scale is a normal float (none zero or subnormal). The quantizer wrote the model into the output file's memory map: 18 minutes and a 46.2 GB peak memory footprint.
• Metadata: the template (4,749 bytes) is Qwen's chat format with thinking off: an empty think block at the generation prompt, past assistant turns with their think block as Qwen's template writes them with preserve_thinking at its default, and Qwen's tool-call layout. Rendered with minijinja 2.14.0 set up as the runtime sets it up (LiteRT-LM's own template rewrites included), it gives byte for byte what transformers gives with Qwen's template at enable_thinking=False in nine conversation shapes (a single question, with a system message, two and three turns, a system message and three turns, past turns with reasoning, a tool list, a tool response, and the runtime's own tool-response part); it does not follow preserve_thinking=False (past think blocks are always kept). No thought channel, supports_thinking false, supports_function_calling true and the tool-call format of LiteRT-LM v0.19.0's models/qwen3_5 configuration (since 2026-11-03); no start token; stop ids 248044 (<|endoftext|>) and 248046 (<|im_end|>), both already declared by the exporter (the repo's generation config lists both as eos); generic_model; fp32 GPU activations declared; max_num_tokens 4096. The tokenizer section is the repo's tokenizer.json byte for byte: the exporter's bundle had carried the tokenizer as transformers writes it for the tokenizer class, which tokenizes Hindi, Thai, Tamil and Arabic with harakat differently (49, 24, 29 and 30 tokens on four test strings against the repo file's 36, 12, 16 and 29); the runtime's tokenizer matches the tokenizers library on the repo file in 292 of 292 strings.
• Not in this file: the vision tower and the multi-token-prediction head (on litert-lm 0.18.1 a smaller Qwen3.5 file with such a head did not build a Mac GPU engine, and on the CPU the turn's stop token is not committed with speculative decoding on, LiteRT-LM issue 3439).

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.

License and changes

Apache-2.0, inherited from Qwen/Qwen3.8-27B (the repo's metadata and its LICENSE file, "Copyright 2026 Alibaba Cloud"); this repository carries a copy of that LICENSE. Changes from the original work: the text decoder of the multimodal checkpoint exported (the vision tower and the multi-token-prediction head not loaded), weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above with the 96 decay/gate projections float; the chat template replaced by a thinking-off form of Qwen's template; the tokenizer repackaged unmodified; stop ids 248044 and 248046, fp32 GPU activations and a 4096-token KV budget declared. This repository is a community conversion and is not affiliated with Qwen or Alibaba Cloud.