Measured on device (edge-compat, lfm2.5-230m-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 41.6 tok/s · prefill 1101 tok/s · TTFT 210 ms · all 492 ops delegated (2026-09-01); iPhone 17 Pro · LiteRT-LM 0.16.0 · GPU · decode 161.7 tok/s · prefill 3036 tok/s · TTFT 71 ms (2026-09-01); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 555.3 tok/s · prefill 12862 tok/s · TTFT 22 ms (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-230m-int4/CARD.md
Measured on device (edge-compat, lfm2.5-230m-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 121.7 tok/s · prefill 3813 tok/s · TTFT 60 ms · all 492 ops delegated (2026-09-01); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 39.4 tok/s · prefill 846 tok/s · TTFT 270 ms · all 492 ops delegated (2026-09-05); iPhone 17 Pro · LiteRT-LM 0.16.0 · GPU · decode 143.2 tok/s · prefill 3422 tok/s · TTFT 75 ms (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-230m-int8/CARD.md
LFM2.5-230M — LiteRT-LM
LiquidAI/LFM2.5-230M converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Tested on litert-lm 0.16.x; the bundles carry the ExecutorMetadata section that litert-lm ≥ 0.15 requires for this hybrid architecture.
LFM2.5-230M is Liquid AI's smallest instruction-tuned model (230M parameters, 8 gated short-conv blocks + 6 GQA attention blocks, tied embeddings, 65k vocab). Liquid positions it for data extraction and lightweight on-device agentic pipelines, not for math or heavy reasoning — our measurements below agree.
| File | Recipe | Size |
|---|---|---|
LFM2.5-230M_int8.litertlm |
int8 dynamic on linears + embedding, convs float | 266 MB |
LFM2.5-230M_int4.litertlm |
int4 blockwise-32 + OCTAV on linears, int8 embedding | 177 MB |
int8 is the recommended file. It is closer to bf16 on instruction following, and at this model size it is also the faster file on Android GPU (2.8× faster decode than int4 on the Galaxy S26) and equal on desktop. int4 is 89 MB smaller and decodes 13% faster on iPhone Metal — pick it for size or iPhone-only deployments.
Correctness
8-question sanity check (Apple M4 Max, litert-lm 0.16.0 lineage): int8 7/8 on CPU and 7/8 on GPU; int4 6/8 on CPU and 8/8 on GPU, no degeneration in any run. The bf16 PyTorch reference scores 8/8 on the same questions. int8's one miss is a rhyme-completion line; int4's CPU run misses that plus one translation question — a real but small quantization cost.
Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno): both files generate correctly on GPU and CPU, with full OpenCL delegation — every prefill signature and the decode graph replaced by LITERT_CL with zero rejected ops (492/492 nodes on each prefill signature).
iPhone 17 Pro: both files pass the same generation check on Metal and CPU (clean stop, thermal nominal throughout).
Accuracy
The model's headline skill is instruction following, so the quantization was chosen on it. We ran the IFEval prompts through a re-implementation of the mechanically checkable instruction types (n=120, prompt-level strict, greedy). These numbers are comparable only within this table — the checkers are simplified, so do not compare them to published IFEval scores:
| Configuration | IFEval-style strict | GSM8K (n=100) |
|---|---|---|
| PyTorch bf16 (reference, MPS) | 60.0% | 23% |
| LiteRT int8 | 58.3% | 23% |
| LiteRT int4 (block-32) | 53.3% | 22% |
int8 is statistically indistinguishable from bf16 on paired prompts. int4 costs ~7 points of instruction following — that is the price of the 89 MB saving. GSM8K is near floor for every configuration, as Liquid's own model card implies; no quantization changes it.
One recipe note: the obvious export-time int8 recipe (which also quantizes the conv layers) measured 53.3% on the same harness — 5 points below shipping int8. These bundles keep the convs float, matching what the LFM2.5-1.2B-JP and 2.6B conversions found for this architecture.
Usage
litert-lm run ./LFM2.5-230M_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
# GPU
litert-lm run ./LFM2.5-230M_int8.litertlm --backend gpu --cache no --prompt "..."
The bundle embeds the model's own chat template (ChatML role markers, tool-calling branch included) and declares stops <|im_end|> and <|endoftext|>. KV budget is 4096 tokens with 11 prefill signatures (1–1024).
One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's short-convolution state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file answered the capital question with "capital" on the same Engine against "Paris" on a fresh one, in both of its runs (a probe is run a second time only when its first run does not pass). The probe could not measure the _int4 file: its fresh-Engine baseline did not pass the probe's own check (all three expected answers present: Paris, 391, blue) in either of its two runs, so there was nothing to compare. Create a new Engine for each conversation. The first conversation on an Engine is unaffected. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05). No release carries it yet (v0.19.1 does not); this note will be updated when one does.
Tool calls on litert-lm 0.18.0. Since the 2026-10-14 re-pack (metadata only: llm_model_type lfm2), the runtime parses the LFM2 tool call and the tool result reaches the model; the earlier files returned the call as message text (<|tool_call_start|>[get_weather(location="Paris")]<|tool_call_end|>). The lfm2 model type needs litert-lm ≥ 0.15.0; the tool-call parser is 0.18.0's — on 0.15–0.17 the call still arrives as text (read from the runtime source, not run here). The 2026-10-14 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous files and have not been re-run — the Mac GPU digit check passed on 0.18.0.
History think blocks: a template property, not observed. This file's chat template keeps a <think>…</think> block only on the assistant turn that follows the latest user message and drops it from earlier turns, while the runtime requires each re-render of the conversation to extend the previous one (my open report). So if a reply ever carried a think block, the second message of that conversation would fail with INTERNAL: The new rendered template string does not start with the previous rendered template string. The model does not open its answers with a think block, and none of the replies in my litert-lm 0.18.0 regression run (Mac, CPU backend, Python API, both files) carried one: the two-turn check answered "Kenji" to "What is my name? Answer with the name only." after "My name is Kenji." on both files. Nothing to do unless you see that error; then keep one message per conversation.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine, ≥300 s rest before each GPU reading:
| File | Backend | Prefill (256) | Decode | TTFT |
|---|---|---|---|---|
| int8 | GPU | 12810 tok/s | 561 tok/s | 0.022 s |
| int8 | CPU | 1727 tok/s | 154 tok/s | 0.155 s |
| int4 | GPU | 12862 tok/s | 555 tok/s | 0.022 s |
| int4 | CPU | 1613 tok/s | 149 tok/s | 0.166 s |
Galaxy S26 (SM-S942Q, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown):
| File | Backend | Prefill (205) | Decode | Peak RSS |
|---|---|---|---|---|
| int8 | GPU (OpenCL) | 3813–3815 tok/s | 121.7–121.9 tok/s | 444 MiB |
| int8 | CPU | 1006–1140 tok/s | 94.7–95.4 tok/s | 636–706 MiB |
| int4 | GPU (OpenCL) | 1101–2476 tok/s | 41.6–43.6 tok/s | 406 MiB |
| int4 | CPU | 262–263 tok/s | 50.7–53.9 tok/s | 538 MiB |
The int4 GPU prefill range's low end is the cold shader-compile run. Note the ordering: at 230M, int8 decodes 2.8× faster than int4 on this GPU — the blockwise dequant overhead dominates at this size. The larger LFM2.5 models show the reverse.
iPhone 17 Pro (BenchmarkApp yardstick, 128-token prompt, cold runs, one reading per cell):
| File | Backend | Prefill (128) | Decode | TTFT | Peak resident |
|---|---|---|---|---|---|
| int8 | Metal GPU | 3422 tok/s | 143 tok/s | 75 ms | 610 MB |
| int8 | CPU | 1530 tok/s | 87 tok/s | 107 ms | 941 MB |
| int4 | Metal GPU | 3036 tok/s | 162 tok/s | 71 ms | 650 MB |
| int4 | CPU | 808 tok/s | 94 tok/s | 206 ms | 791 MB |
iPhone Metal is the one backend where int4 out-decodes int8 (162 vs 143 tok/s).
Conversion notes
Converted with litert-torch 0.9.3 / transformers 5.14.1 / ai-edge-quantizer, released wheels only (reproduction scripts: hf-to-litertlm).
- The upstream chat template does not run on the runtime's Jinja engine. LiquidAI's
chat_template.jinjauses HF's{% generation %}/{% endgeneration %}training-mask markers (a parse error in minijinja) andmessage.get("content")(minijinja maps have no.getmethod). The embedded template strips the two markers and rewrites the two.getcalls to plain indexing. Rendering is byte-identical to the original across user/system/multi-turn/tool-calling conversations, verified through HFapply_chat_template. - KV budget 4096, not 4099. With the 1024-token prefill signature present, a 4099-token KV cache fails GPU engine creation at shader compile (measured on macOS WebGPU); 4096 — a multiple of the widest signature — compiles everywhere we tested.
- Convs stay float in the int8 file (see Accuracy). int8 is applied post-export to linears + embedding.
ExecutorMetadataadded post-export (the released exporter omits it for this state-carrying architecture; litert-lm ≥ 0.15 needs it to bind the 8 conv states and 12 KV buffers). Weights are byte-identical through the edit.- Stops declared as token ids 7 (
<|im_end|>) and 2 (<|endoftext|>); BOS<|startoftext|>. Tokenizer embedded as the upstreamtokenizer.json.
Known runtime issues
External weights (since 2026-11-09). In LFM2.5-230M_int4.litertlm and LFM2.5-230M_int8.litertlm, the weights sit in a TFLiteWeights section at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier files with the weights inside the graph stay at revision 7352022.
Long greedy replies on the GPU may vary between runs. On the GPU's default path, where the sampler writes the inputs of the next decode step, a long greedy reply of LFM2.5-2.6B differed from one process to the next on a Mac (WebGPU, LiteRT-LM 0.19.1 from PyPI and built from source) and on a Pixel 11 Pro GPU (PyPI litert-lm-api 0.19.0); with --sampler_handles_input=false (litert_lm_advanced_main built from source; the Python, Kotlin and Swift APIs of 0.19.1 have no such setting) it gave one reply. The files here run the same GPU path; this was not measured on them.
On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: these files carry stop sequences of <|im_end|> and a newline with each of 13 characters in front (12 punctuation marks and a space), so the runtime holds the mark back to see whether <|im_end|> and a newline follow, the answer ends on <|im_end|> alone, and the held mark is never written. LiteRT-LM 0.17.0 and later write the held mark (StopTokenStreamFilter in runtime/core/tasks.cc, read in the source of 0.17.0 to 0.18.1). Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop itself was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle whose stop sequences have the same form.
From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
Structured output: LiteRT-LM refuses constrained decoding for files of llm_model_type lfm2 when the conversation is created (Constrained decoding is not supported for Lfm2DataProcessor.), so such a file does not take a JSON ResponseFormat. That was measured on LFM2.5-1.2B-Instruct_int4.litertlm from LFM2.5-1.2B-Instruct with litert-lm 0.16.0 and 0.19.1 on a Mac CPU. Both files here carry llm_model_type lfm2; they were not run with constrained decoding.
On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error.
License and changes
Distributed under the LFM Open License v1.0 (inherited from the base model; see the license link above). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and a render-equivalent repair of the chat template repackaged into the .litertlm bundle; runtime metadata (stops, ExecutorMetadata) added. This repository is a community conversion and is not affiliated with Liquid AI.