Measured on device (edge-compat, spark-x2.5-4b-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 4.9 tok/s · prefill 56 tok/s · TTFT 3.98 s · all 2233 ops delegated (2026-09-07); Mac Studio M4 Max · LiteRT-LM 0.17.0 · GPU · decode 53.6 tok/s · prefill 826 tok/s · TTFT 329 ms (2026-09-07); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 5.5 tok/s · prefill 35 tok/s · TTFT 6.31 s (2026-09-07). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/spark-x2.5-4b-int4/CARD.md

Measured on device (edge-compat, spark-x2.5-4b-int8): Mac Studio M4 Max · LiteRT-LM 0.17.0 · GPU · decode 49.4 tok/s · prefill 784 tok/s · TTFT 347 ms (2026-09-07); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 4.5 tok/s · prefill 65 tok/s · TTFT 3.49 s (2026-09-07); Mac Studio M4 Max · LiteRT-LM 0.17.0 · CPU · decode 19.5 tok/s · prefill 345 tok/s · TTFT 792 ms (2026-09-07). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/spark-x2.5-4b-int8/CARD.md

Spark-X2.5-4B — LiteRT-LM

XHToken/Spark-X2.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Tested on litert-lm 0.17.0.

Spark-X2.5 is the SparkLLM Team's (iFLYTEK) open model series released in August 2026 — a 4.11B dense decoder (36 layers, hidden 2560, 16 query / 4 KV heads of 256, sliding-window(512) x3 : full x1, 131,072-entry tied vocab) with a hybrid attention layout of three sliding-window layers per full-attention layer, per-head sigmoid attention-output gates, exact-GELU gated MLPs and a native 1M-token context in the original. It is a reasoning model: it works problems inside <think>…</think> before answering, and these bundles carry that machinery — the generation prompt pre-fills the think opener exactly as the vendor chat template does, and a thought channel is declared in the bundle metadata so the runtime separates reasoning from the answer and honours a thinking budget. Apache-2.0.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Both files are at bf16 parity on GSM8K (94 = 94 = 94) — with fp32 GPU activations declared in the bundle (under the runtime's default fp16 GPU activations the same int8 weights scored 84; see Conversion notes). int4 (block-128) is the recommended file: same accuracy, 41% smaller (2.48 GB), and the faster GPU file on the Mac (53.6 vs 49.4 tok/s decode). int8 (4.24 GB) is the CPU-prefill file (345 vs 118 tok/s on the Mac CPU). On the Galaxy S26 this 4B is a ~5 tok/s model on every measured path — the fp32-activation OpenCL path does not beat the CPU there (int4: GPU 4.9 vs CPU 5.5–5.6 tok/s decode) but halves peak memory (2.0 vs 3.5 GB); pick by memory, not speed.

Correctness

• 8-question sanity gate (Apple M4 Max, litert-lm 0.17.0 CLI, one process per question, greedy, thinking on): int8 CPU 8/8 / GPU 8/8, int4 CPU 8/8 / GPU 8/8; the bf16 PyTorch reference scores 8/8 on the same questions through the vendor chat template. Every answer arrived as a clean final answer through the thought channel — no reasoning leakage, no degeneration, every think block closed.
• Tokenizer parity: the bundle's tokenizer section (the upstream tokenizer.json, byte-level BPE) encodes 234 of 234 probe rows — every added token alone and mid-string, the role-marker turn, Latin-1 / Extended-A, emoji, CJK, digits, whitespace — to the same ids as the tokenizers reading of the upstream file.
• Multi-turn (3 turns, Mac CPU, python API, greedy: plant a fact, do arithmetic, recall the fact; the same conversation run twice on one engine, both times at the runtime's default, which drops each turn's thought from the KV cache before the next turn): pass on both files; no <think> / </think> in any answer.

Accuracy — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, 3584 output tokens (a reasoning model truncated mid-thought never reaches its answer; 2048 cut a third of the bf16 model's first questions), n=100, identical prompt, harness and answer extraction for every row; the LiteRT rows ran on the litert-lm 0.17.0 engine (GPU backend), scored on the text after the thought channel:

Usage

litert-lm run ./Spark-X2.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word." --thinking true

# GPU
litert-lm run ./Spark-X2.5-4B_int8.litertlm --backend gpu --cache no --thinking true --prompt "..."

Notes for a reasoning model:

• Give it a generous output budget (≥ 2048 tokens, 3584 for math). The model thinks before it answers; truncated mid-thought it produces no final answer at all.
• The bundle declares the thought channel (<think> … </think>), so runtimes that expose ThinkingConfig / a thinking budget can cap or read the reasoning separately; the streamed answer contains only the final response. enable_thinking: false renders the vendor's no-think form (<|Bot|></think>).
• The bundle carries the vendor prompt format as a Jinja template: a default system block (<|System|>\nyou are a helpful assistant., a user system prompt appended after it), every message wrapped in <｜start▁of▁sentence｜> … <｜end▁of▁sentence｜>, generation prompt <|Bot|><think>; stop token <｜end▁of▁sentence｜>. Tool-call formatting is not carried. KV budget 4096 tokens (the original's 1M context does not apply on-device), six prefill signatures (1024, 256, 64, 16, 4, 1) — six rather than eleven because every exported signature is charged engine memory whether or not it is called.
• The vendor's recommended sampling is temperature 1.0 / top-p 0.95; the numbers on this card are greedy.
• Replies that take over 10 minutes. On litert-lm 0.18.x and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower max_output_tokens or use the streaming send_message_async, which has no such limit.

Performance

Apple M4 Max (litert-lm benchmark 0.17.0, -p 256 -d 256 --runs 3 --cache no, quiet machine, serialized, ≥300 s rest before each GPU reading; every backend first proven to generate on the file):

Init is the Init time the CLI prints; that value includes both Init Total and the phases inside it, so it counts engine creation twice (LiteRT-LM #4057).

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown):

iPhone 17 Pro (G41DeviceTest harness, composite 8-question prompt, --max-tokens 3072, on-device byte count verified against the source file; score = questions answered correctly inside one reply):

Both files run with fp32 GPU activations on the phone as well. The int8 file is a single 4.24 GB weights section and loads under this harness (Increased Memory Limit entitlement; available memory floor ~2.6 GB on the Metal leg, ~4.5 GB on the CPU leg); the int4 file's Metal leg bottomed at ~2.0 GB free. GPU init times are with a warm shader cache (a cold first launch of the int8 file initialised in 18.8 s). The int4 file's one miss on each backend is the rhyme line at the end of the composite 8-question prompt (it answers "sweet"); asked on its own the same question is answered "blue" on every backend.

Conversion notes

Converted with litert-torch 0.9.3 / transformers 5.14.1 / ai-edge-quantizer, released wheels (reproduction script: hf-to-litertlm).

• The vendor modeling code is patched for export, not re-implemented. modeling_spark.py computes attention through its own eager function and ignores config._attn_implementation; the export copy dispatches through the registered attention interface (so litert-torch's transposed KV cache is used), threads the per-call kwargs, declares the attention-backend capability flags, and applies the per-head sigmoid output gate in the interface's layout. In eager mode the patched file is bit-identical to the vendor file (max |Δlogit| 0.0 on 24 random tokens); two further edits make the vendor file load under transformers 5 at all (_tied_weights_keys mapping form, mask-utility kwargs).
• The think opener is pre-filled by the bundle's template and the thought channel is declared in the metadata. Without the channel the runtime streams raw reasoning into the answer and silently ignores any thinking budget.
• No start token in the metadata. The tokenizer declares <｜start▁of▁sentence｜> as BOS but never prepends it (add_bos_token: false); the template carries its own. The exporter's unconditional start_token write was suppressed (measured harmless in bf16 on the 8-question gate, but it is not the vendor prompt).
• int4 externalizes the embedding table. The vocab is tied; asking int4 for the lm_head and int8 for the embedding makes the quantizer copy the 131,072-row table once per signature (measured 4.6 vs 1.7 bytes/parameter on a tiny checkpoint). The embedder lives in its own section instead. The int8 file needs no split.
• GPU activations are declared fp32 in the bundle (prefer_activation_type = "fp32", a metadata-only repack; weights byte-identical). The runtime runs the text decoder with fp16 activations on the GPU by default, and on this 36-layer decoder that silently costs reasoning accuracy: the same int8 weights scored GSM8K 84 under fp16 GPU activations and 94 under fp32 — established by re-running the GPU losses on the CPU (fp32) path, where 10 of 11 came back right. The 8-question gate did not see it (8/8 either way); only a long-generation accuracy row does. Expect a larger GPU memory footprint than an fp16-activation file of the same size.
• Tokenizer embedded as the upstream tokenizer.json (HF tokenizer section), not a SentencePiece conversion.
• Quantization at export time: int8 = dynamic per-channel on linears + embedding; int4 = int4 blockwise-128 + OCTAV on linears, int8 embedding (externalized). No post-processing.

Known runtime issues

External weights (since 2026-10-30). In Spark-X2.5-4B_int4.litertlm and Spark-X2.5-4B_int8.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier files with the weights inside the graph stay at revision 08c922b.

With thinking off, from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because Spark-X2.5-4B_int4.litertlm holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). With thinking on, the runtime rebuilds the cache from the re-rendered history at each turn, so the second turn is free of it when the conversation opens with a user message; the same source shows a zero embedding at another token from the third turn on (not measured). Spark-X2.5-4B_int8.litertlm has no embedder section (the decoder looks the tokens up itself) and is not affected. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: these files carry <｜end▁of▁sentence｜> with each of 13 characters in front (12 punctuation marks and a space) as stop sequences next to <｜end▁of▁sentence｜> alone, so the runtime holds the mark back to see whether <｜end▁of▁sentence｜> follows, and when the answer ends there the held mark is never written. From LiteRT-LM 0.17.0 on (read in the source of 0.17.0 to 0.18.1) Spark-X2.5-4B_int4.litertlm still drops 12 of the 13 marks and keeps only the space, and Spark-X2.5-4B_int8.litertlm drops all 13: the mark and <｜end▁of▁sentence｜> complete two stop sequences at the same token, <｜end▁of▁sentence｜> alone and the one with the mark, the runtime records the one that comes later in the file's stop list (for <｜end▁of▁sentence｜> alone, its first entry when the file lists it twice), and it writes the held mark only when that is <｜end▁of▁sentence｜> alone. Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop on releases before 0.17.0 was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle with punctuated stop sequences.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.

License and changes

Distributed under Apache-2.0 (inherited from the base model; the upstream LICENSE file is included). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle (thinking pre-fill and thought-channel metadata as described); tool-calling template branches not carried. No training or fine-tuning. This repository is a community conversion and is not affiliated with iFLYTEK or the SparkLLM Team.