Qwen3.5-0.8B MTP — LiteRT-LM speculative-decoding bundle (bug-repro artifact)

This is a test artifact, not a release. It is published so that LiteRT-LM issue #3439 (under --speculative-decoding true the runtime never commits the turn-final stop token) can be reproduced with a ready-made bundle. Single-turn generation and benchmarks work; multi-turn conversations under the flag see a transcript with every assistant turn's <|im_end|> missing, and this 0.8B visibly degrades on later turns. For a Qwen3.5-0.8B to build on, use litert-community/Qwen3.5-0.8B.

Qwen/Qwen3.5-0.8B converted to .litertlm with a verify signature, plus the checkpoint's own 1-layer MTP (multi-token prediction) head packed as the drafter section LiteRT-LM's speculative decoding consumes (tf_lite_mtp_drafter). Weights are dynamic-range int8 on the linears and embedding tables; cache length 4096; three draft tokens per round. Built with the open recipe in hf-to-litertlm/mtp_work (litert-torch at 115a136 plus the patch in that directory, litert-lm 0.16.0 CLI for packing).

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Run

pip install litert-lm            # 0.16.0 or 0.16.1
litert-lm run Qwen3.5-0.8B_mtp_drafter_int8.litertlm --backend cpu --speculative-decoding true \
    --prompt "Explain photosynthesis in three sentences."
litert-lm benchmark Qwen3.5-0.8B_mtp_drafter_int8.litertlm --backend cpu --speculative-decoding true --cache no

The drafter logs MTP Drafter - Success rate (accepted / drafted) when the engine shuts down.

Reproduce #3439

python stop_token_probe_3439.py Qwen3.5-0.8B_mtp_drafter_int8.litertlm

The probe runs one 5-turn greedy conversation with the flag off and on and prints, per turn, the engine's committed decode-token count, that turn's prefill size, and the conversation's token_count. Measured on macOS (Apple M4 Max, CPU) with litert-lm 0.16.0 on 2026-09-03 and 0.16.1 on 2026-09-04, identical output:

--- enable_speculative_decoding=False
'MANGO'                  decode_committed=3   prefill=21  token_count=24
'KIWI'                   decode_committed=3   prefill=21  token_count=48
'MANGO'                  decode_committed=3   prefill=30  token_count=81
'MANGO'                  decode_committed=3   prefill=18  token_count=102
'MANGO'                  decode_committed=3   prefill=21  token_count=126
--- enable_speculative_decoding=True
'MANGO'                  decode_committed=2   prefill=21  token_count=23
'MANGO'                  decode_committed=2   prefill=21  token_count=46
'MANGO'                  decode_committed=2   prefill=30  token_count=78
'M'                      decode_committed=1   prefill=18  token_count=97
'M'                      decode_committed=1   prefill=21  token_count=119

Flag off commits reply + stop (3 tokens for "MANGO"); flag on commits 2, the next prefill is the same size, so nothing re-adds the token and token_count drifts by −1 per turn. A 2B bundle built the same way shows the same drift with the text intact.

Performance (measured)

macOS, Apple M4 Max, CPU, litert-lm 0.16.0 CLI, benchmark -p 128 -d 256 --cache no, 2026-09-01:

Engine acceptance on four prompts: 0.48–0.71 per draft token, i.e. 2.4–3.1 committed tokens per round (mean 2.76). On a Galaxy S26 (CPU, capped 150-token legs, interleaved with cooling waits, median of 3) the flag-on leg was slower on document text: 32.4 → 22.9 tok/s. On iPhone it was slower still in my harness. So this bundle is a mechanism proof and a bug repro, not a speed recommendation.

Correctness record

• Interpreter-level gate against the stock HF fp32 model with forced rejection at every draft index and a continuation prefill over the rejected tail: 33/33 checks, logits max|diff| ≤ 6e-5, argmax equal at every committed position.
• Released runtime (0.16.0, CPU): 8-question chat gate 7/8; single-word fills 1..16 under the flag 16/16.
• Output vs the same bundle with the flag off: 1 of 4 prompts byte-identical; the rest diverge at int8 near-ties (HF-fp32 referee top-2 gap 0.125–0.375 at every divergence), not at a contract error.

Limitations

• CPU only. The GPU backend was not gated with the drafter section.
• Multi-turn use under the flag hits #3439 until the runtime fix lands; flag off, the bundle behaves as a plain Qwen3.5-0.8B int8 conversion with a simple ChatML template (thinking disabled).
• One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file answered the arithmetic question with 491 on the same Engine against 391 on a fresh one in its first run (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue). In the second run both Engines gave all three expected answers. In both runs the file answered the capital question with a sentence ("The capital of France is Paris.") on the same Engine against the one word "Paris" on a fresh one. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.
• CPU digit test. "What is 17 times 23?" at the default rendering (thinking disabled by this bundle's template), drafter flag off, 3 fresh CPU processes: the _int8 file answered 391 in 3 of 3 with no repetition (53, 53 and 48 output tokens against a 400-token cap; litert-lm 0.18.0, Mac CPU). In a separate CPU probe (the weather question with a tool supplied), one of its two runs repeated a Wind / Humidity / Visibility list until the 400-token cap, and the other answered in three sentences. With the flag off this file behaves as a plain Qwen3.5-0.8B int8 conversion that keeps the decoder recipe of the litert-community/Qwen3.5-0.8B int8 file published until 2026-10-12 (the delta-rule decay/gate projections int8; that repo re-quantized its file that day). The check is here because the sibling Qwen3.5-2B int8 file looped on 17 times 23 on CPU until its 2026-10-11 re-quantization (details on its card).
• 8 GB-class phones and Mali. The GPU backend has not been run with this file on any device. iOS (Metal) and Android GPU: not verified with this file; the GPU program cache growth per engine creation is not measured with this file either. On Android, no Mali phone has run this file. Its GPU engine is not expected to fit an 8 GB-class phone — an expectation carried over from the Qwen3.5-0.8B int8 file published until 2026-10-12, whose decoder recipe this file keeps. On an 8 GB Pixel 8a (Arm Mali, MemTotal 7,754,700 KiB), that file's GPU engine creation reached 3 of the 11 prefill signatures before free memory fell below 400,000 KiB, the test's stop line (the reading that stopped it was 361,000 KiB). Whether that build completes without the stop is not measured. Use the CPU backend on 8 GB phones, as on every device for this file (the only backend it has been run on).
• Tool calls. This bundle's chat template has no tools section, so tool definitions never reach the prompt. On litert-lm 0.18.0 the runtime also parses no Qwen3.5 tool call: such a call comes back as plain text and tool_calls stays empty (my open report). With a tool supplied and the runtime's automatic tool calling turned on, the _int8 file wrote a weather answer of its own instead of calling the tool (tool_calls stayed empty), in each of two runs — one of them the run that repeated a list until the cap (litert-lm 0.18.0, Mac CPU).
• Not listed in any catalog and not maintained as a release; it will be updated or removed when #3439 is resolved.

Known runtime issues

From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.

Mac GPU, long outputs: not verified on this file. On a Qwen3.5-4B build (int8 and Mixed INT4) on an Apple M4 Max, the default GPU path, where the sampler also writes the next decode step's inputs (the log says Sampler will handle decode input tensors.), gave a different greedy reply on each run and emitted tokens the model itself scores at a log probability of −16 to −23 (litert_lm_advanced_main built from LiteRT-LM main at dab2e708); with litert-lm 0.19.0 from PyPI, which always takes this path, 16 of 22 long code replies failed node --check. With --sampler_handles_input=false, or --sampler_backend=cpu, on that C++ CLI the reply was the same on every run and 0 of 7 failed. The Python, Swift, Kotlin and C APIs of 0.19.0 have no setting for this.