DeepSeek-R1-Distill-Qwen-7B — LiteRT-LM (blockwise int4)
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
converted to the LiteRT-LM (.litertlm) format for on-device inference with
Google's LiteRT-LM runtime (the
engine behind the official litert-community/* models).
A reasoning model: it emits a <think> … </think> chain before the answer.
MIT-licensed (distilled onto an Apache-2.0 Qwen2.5 base). Converted with the
official upstream litert-torch — no fork, no custom code.
| File | model.litertlm (~4.2 GB) |
| Quantization | int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8 |
| Compute | integer |
| Context (KV cache) | 4096 |
| Base model | deepseek-ai/DeepSeek-R1-Distill-Qwen-7B |
| Decode speed | ~67 tok/s (Mac M-series, LiteRT-LM, Metal GPU, greedy) |
| Platforms | Desktop (Mac) ✓ · high-RAM (12 GB+) Android ✓ · iPhone / 8 GB phones ✗ (4 GB exceeds the budget) |
Usage
litert_lm_main --model_path model.litertlm --backend gpu \
--input_prompt "If a train travels 60 km in 45 minutes, what is its speed in km/h?"
The .litertlm bundle carries the tokenizer and the DeepSeek prompt template
(<｜User｜> / <｜Assistant｜>, stop token <｜end▁of▁sentence｜>). The assistant
opens a <think> block, reasons step by step, then gives the final answer
(commonly in \boxed{}).
Thinking channel. Since the 2026-10-14 re-pack (metadata only: LlmMetadata.channels declares the thought channel with <think>…</think>), the reasoning arrives on the thought channel instead of the message text, as in litert-community/DeepSeek-R1-Distill-Qwen-7B. R1-style reasoning runs long: with a small max_output_tokens (100–200 in our suite) the answer text stays empty while the model is still reasoning — give it 2048 or more.
Replies that take over 10 minutes. On litert-lm 0.18.x and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower max_output_tokens or use the streaming send_message_async, which has no such limit.
Run on desktop (LiteRT-LM CLI)
The same .litertlm bundle runs on macOS / Linux / Windows with the official
LiteRT-LM CLI — including as a
local OpenAI-compatible API server:
pip install litert-lm
litert-lm import --from-huggingface-repo mlboydaisuke/DeepSeek-R1-Distill-Qwen-7B-LiteRT model.litertlm deepseek-r1-distill-qwen-7b-litert
litert-lm run deepseek-r1-distill-qwen-7b-litert # interactive chat in the terminal
litert-lm serve # local OpenAI-compatible API server
Quality — GSM8K parity
GSM8K (n=100, greedy, 0-shot, identical prompt + answer-extraction; max_new_tokens=2048
to fit the reasoning chain).
| Configuration | GSM8K |
|---|---|
| bf16 (reference) | 88.0% |
| This model — LiteRT int4 (BOCTAV4) | 87.0% |
LiteRT int4 is at parity — −1.0 pt vs bf16. The reasoning behavior is fully preserved through 4-bit quantization; the shallow-wide Qwen2 (28 layers) absorbs int4 rounding cleanly.
Conversion
Converted with the official upstream litert-torch
export_hf (clean git worktree at upstream/main, dev-fork patches excluded).
Qwen2ForCausalLM rides the stock converter with no custom code. int4 recipe =
blockwise (block 32) + OCTAV with INT8 embedding (externalized into its own
bundle section); KV cache 4096.
License
MIT (model weights), inherited from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B; the Qwen2.5 base is Apache-2.0. Commercial use and derivatives permitted.
Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.