LFM3-Omni-2B — LiteRT

LiquidAI/LFM3-Omni-2B reads text, pictures, video and audio and answers in text or speech. This repository is the LiteRT form for text, picture and audio input: one .litertlm file that answers in text from text, picture and audio input with Google's LiteRT-LM runtime, plus LiteRT graphs and a Python runner for speech output. On a Galaxy S26 GPU with LiteRT-LM 0.18.1 the .litertlm file decodes at 27.0 tokens/s and gave the same answers as on a Mac for all 14 test prompts (October 28, 2026).

Converted from LiquidAI/LFM3-Omni-2B at revision ce2458cb (2.17 B parameters, BF16). The provider also publishes GGUF (LiquidAI/LFM3-Omni-2B-GGUF); this repository is the LiteRT form.

What runs where

Files

Quick start

pip install -r requirements.txt
python examples/run_omni.py "What is the capital of Japan? Answer briefly."
python examples/run_omni.py --image photo.png "What animal is in this picture? Answer briefly."
python examples/run_omni.py --audio speech.wav --system "Perform ASR in English."
python examples/run_omni.py --speech tts "The quick brown fox jumps over the lazy dog." --out fox.wav
python examples/run_omni.py --speech interleaved "What is the capital of France?" --out answer.wav
python examples/run_omni.py --fixtures   # runs fixtures/fixtures.json with its checks

Recordings are 16 kHz mono 16-bit WAV. A recording longer than 30 s is read in 30 s windows for a text answer as long as it fits the 4,096-token context (a 251 s recording did, a 502 s one did not); speech output takes text or one recording of 30 s or less. The example creates a new Engine for each question. The speech loop runs on the CPU.

Performance

Galaxy S26 (SM-S942Q), LiteRT-LM 0.18.1, 1,024 prompt tokens, 256 generated tokens, the file's 4,096-token context, median of 2 processes.

With fp32 activations on the same GPU the decoder runs at 270 / 23.5 tok/s.

Speech output on a Mac M4 Max CPU: 0.48–0.50 s of compute per second of audio for the decoder, depth decoder and frame embedding; the codec adds about 6 ms per 0.96 s of audio on the same CPU (4 threads). In the Android sample photo_voice_chat (Kotlin, LiteRT 2.3.0, airplane mode), the language model and the other parts run on the CPU (4 threads); each answer below was started on a cooled phone (October 31, 2026). On a Galaxy S26, making one second of speech took 0.75 and 0.75 s with depth_frame2 on the GPU at float16 (the sample's choice on phones with a Qualcomm chip): two answers of 7.1 s, neither paused, the first sound 2.0–2.2 s after the end of the spoken question; three answers in a row at the end of the session, with no cooling between them, took 0.81, 0.79 and 0.80 s. With depth_frame2 on the S26's CPU it took 0.99 and 0.92 s (two answers of 12.4 s). On a Pixel 11 Pro it took 1.62 and 1.58 s with depth_frame2 on the CPU (the sample's choice on other phones; the speech paused 5 times in each 7.2–7.3 s answer) and 1.76 and 1.85 s on its GPU at float16. In an earlier session the same day, depth_step three times per frame on the CPU took 0.98 and 0.91 s on the S26 and 1.67 and 1.80 s on the Pixel 11 Pro, against 0.88 and 0.91 s and 1.56 and 1.54 s for an earlier version of depth_frame2 (not in this repository) on the CPU. With depth_step, nine answers in a row on a Galaxy S26 took 1.15 s per second of speech on average (0.88–1.30 s), and the speech paused in 4 of them once the phone was warm. The first sound is when the answer's audio starts to play: the model's speech starts with a near-silent lead (0.7–1.3 s in eight of the sample's answers on the phones, 0.3–1.1 s in nine from the original model on a Mac), so its first word is heard that much later.

Accuracy

Compared with the source model run in float32 (PyTorch, CPU):

• Text answers (15 prompts: 8 questions, 3 pictures, 3 audio clips, one picture plus a spoken question; 14 have a check): every check passes on the Mac CPU and GPU, and on those 14 the answer text equals the source model's. The 15th, a free description of a picture, has no check; its text matches the source model's for the first 8 of 32 tokens.
• The Galaxy S26 GPU and the Pixel 8a GPU gave the same text as the Mac on all 14; the Galaxy S26 CPU on the 10 it ran (the 8 questions, one picture, one clip).
• Next-token agreement with the source model, teacher-forced: 230 of 233 positions.
• Speech output (2 text-to-speech prompts, 2 spoken replies, 3 seeds each): 12 of 12 runs pass the Whisper transcript, length and loudness checks. For one text-to-speech prompt Whisper writes "nine" and "six" as digits (WER 0.13 against the typed text, the same as for the source model's own audio); that prompt is judged against the source model's WER.
• The decoder's weights are rescaled per channel (float32 results stay equal) and then quantized to int8. The mean cosine of its final hidden state to the source model is 0.9958 on the CPU. The source model computed in its own bfloat16 reaches 0.9998 by the same measure.

Notes

• Create one Engine per conversation: on LiteRT-LM, a conversation started on an Engine that already served another one continues from that conversation's running state (LiteRT-LM issue #3165). This model keeps a running short-convolution state.
• From the second message of a conversation, LiteRT-LM 0.18.1 feeds the previous answer's stop token to the model as an all-zero embedding vector. This is a LiteRT-LM 0.18.1 behaviour for every file with an embedder section; it was not measured on this file. One question per conversation avoids it.
• On the CPU, XNNPACK quantizes the int8 layers' activations at run time. Plain dynamic int8 changed one close text answer and the hidden states at picture positions on the CPU, so the decoder's weights are rescaled per channel before the int8 step; with that, the Galaxy S26 CPU answers equal the Mac's on the 10 prompts run.
• Pictures are stretched to 512×512 (the aspect ratio is not kept) and always become 256 image tokens.
• Audio input: exact-zero samples at the end of a clip count as padding, so a clip can give 2–3 fewer audio tokens than the source processor. The measured answers did not change.
• The audio encoder is run on the CPU by the example and in every measurement here.
• The file has no sampler settings: answers are greedy unless your app sets a sampler. For speech, the runner uses the source model's settings (temperature 0.4, repetition penalty 1.2 over the last 64 frames).
• Spoken replies (--speech interleaved, text and speech frames generated in turns): the text can vary with the speech frames sampled before it, as it does in the source model.
• Tool calls come back as message text; your app parses them.
• The LiteRT-LM command-line tool sends a prompt file as one user message, with no system message; use the Conversation API for a system message.
• This file holds 4,096 tokens of context (the source model supports 32K).
• The source model is not a reasoning model; this file declares no thinking channel.
• On the Mac GPU, depth_step at Metal's default float16 speaks as well as at float32: over 90 utterances (30 sentences, 3 seeds) the Whisper word error rate was 0.000 % against 0.117 % at float32, UTMOS22 (a predicted listening score, 1 to 5) 4.468 against 4.478, and the spoken replies to 8 questions (3 seeds) were right 21 of 24 times, as at float32. An earlier version of depth_frame2 (not in this repository) at Metal's float16 stopped some speech early (word error rate 3.404 %): its random draw rounded up to 1 and picked the end of the speech. The Android sample keeps each draw at or below 1 − 2^-11, and depth_frame2 compares the cumulative probabilities with the draw times their total; with both, the sample's speech on the Galaxy S26 GPU at float16 passed the same test as that earlier version on the same phone's CPU (114 answers made on the phone: word error rate 0.000 % for both, UTMOS22 4.481 against 4.471, 21 of 24 spoken replies right for both). On the Pixel 8a and Pixel 11 Pro GPUs, float16 has not been through the speech test. Keep the codec on the CPU: on the Mac GPU at float32 it gives the same words (word error rate 0.117 % for both) with a worse sound (UTMOS22 4.145 against 4.478, lower in all 90 utterances).

License

LFM Open License v1.0, the same license as the source model (LICENSE). The files here are converted from LiquidAI/LFM3-Omni-2B. Section 5 of the license does not license commercial use by an organization with annual revenue of 10 million US dollars or more (a qualified non-profit's non-commercial and research use is exempt).