LFM3-1.7B — TT-native mixed BFP4/BFP8 (GPTQ MLP), single Tenstorrent P150

A community, experimental build of LiquidAI/LFM3-1.7B for one Tenstorrent Blackhole P150 card, served through vLLM with an OpenAI-compatible API: a GPTQ/BFP checkpoint in the lfmq-1 format, a tt-metal runtime with a fused, traced decode step and prompt-lookup speculation, and measured speed and quality.

Not an official Liquid AI or Tenstorrent release. This is a modified derivative of LFM3-1.7B made by Lottolabs. The checkpoint stores Tenstorrent block-floating-point codes (lfmq-1 format) that only the bundled runtime reads: it is not GGUF, AWQ, GPTQ-for-CUDA or a Transformers checkpoint, and it does not load in stock Transformers, llama.cpp or upstream vLLM. Text only.

Every number below cites the file it comes from under evidence/ (the build-tree paths of the result files, kept as they were written). Source paths such as tools/serve_lfm3.sh, lfm3/, plugin/, docker/, quant/*.py and the design notes NOTES-*.md are inside runtime/lfm3-runtime-src.tar.gz. Measured 2026-10-27 – 2026-11-12 on one P150a (host: 32 threads, shared with other teams through a device lock); host load averages are given with the speed tables.

What is in this release (s13-20261112)

Release s13-20261112: what changed since s12-20261030

The checkpoint is unchanged (same files, manifest.json SHA256 cf3ba5e4…); the runtime image is new (dev-s13, NOTES-runtime.md section 18). launch.py keeps the same options; --speculate on (the default) now applies at every --max-num-seqs.

• Prompt-lookup speculation with any number of users (shared verify rounds). s12 speculated only with --max-num-seqs 1. s13 verifies the drafts of every running user in one shared round: one traced step of 2 / 4 / 8 / 16 rows in which each user contributes its pending token plus its drafts (a user without drafts rides as one row). A planner prices each round from a measured device-cost table (evidence/logs/spec13/timing.json) and takes drafts only where they pay (proportional fairness across users, so no user is slowed for the others' throughput); otherwise it runs a plain step. Speculation never changes the output: every reply of the 16-slot gate is identical to the request alone, with decode-ahead off and with speculation off (362/362 each, evidence/logs/serving/dev-s13/summary.txt). Aggregate tok/s (per-user tok/s), 16-slot server, default sampling, median of 3 (evidence/logs/bench/dev-s12/workloads_b16_q.json → evidence/logs/bench/dev-s13/workloads_b16_q.json; every reply's text identical between the two images):

  workload	1 user	2 users	4 users	8 users	16 users
  chat_2k	265.2 → 348.5 (267.3 → 351.1)	519.0 → 602.8 (261.6 → 326.2)	981.2 → 1,001.8 (250.2 → 271.1)	1,753.3 → 1,714.2 (225.9 → 222.9)	2,857.6 → 2,836.4 (187.4 → 185.9)
  code_edit	263.5 → 1,446.2 (264.6 → 1,498.3)	509.4 → 1,653.4 (257.9 → 852.0)	947.3 → 1,842.1 (241.7 → 478.8)	1,647.2 → 1,927.4 (212.0 → 249.2)	2,577.2 → 2,561.1 (167.7 → 166.8)
  tool_write	264.0 → 1,438.0 (266.2 → 1,499.4)	510.0 → 1,610.2 (257.4 → 832.7)	909.0 → 1,760.8 (240.2 → 454.6)	1,577.4 → 1,849.6 (209.6 → 244.3)	2,458.7 → 2,475.7 (164.9 → 164.1)
  gsm8k	266.6 → 328.7 (271.3 → 336.2)	482.6 → 575.6 (267.7 → 317.2)	826.1 → 905.0 (262.3 → 285.8)	1,465.6 → 1,522.2 (246.1 → 248.3)	2,518.6 → 2,475.4 (216.3 → 213.5)

  The gain is where rows are free for drafts (1–4 users, and copy-heavy work up to 8); with 16 users every row is a user, so s13 runs plain rounds and matches s12 within noise. With one user on a 1-slot server s13 equals s12 (evidence/logs/serving/dev-s13/spec_table.md vs evidence/logs/serving/dev-s12/spec_table.md).

• Trace page-table bug fixed; s12 is affected. A decode trace captured later can have its input tensors inside the (released) intermediates of an earlier trace, so every replay of the earlier trace overwrites them. Tokens and positions are re-copied before every launch, but the page table was re-copied only when it differed from the last one staged for that trace, so a step that returned to the same trace with the same pages after another decode trace ran could read KV through garbage block ids (wrong tokens, later NaN logits). s13 keeps only the launched trace's page table as known; the regression test (every decode trace stepped with every other one replayed in between) failed before the fix and passes 10/10 after (evidence/logs/dev/s13b_sw_pre.log, s13b_sw_post.log, evidence/logs/dev/s13b_chk1.log). The same code is in release s12 (s12-20261030): it needs identical pages at the same trace after another trace ran; s12's gates and benchmarks did not hit it, s13's shared-round tests did. Use s13.

• LocalMaxxing matrix, s13: https://www.localmaxxing.com/en/inference-benchmarks/cmwcf0qxy01gdmi01twvma3jz (evidence/lmx/localmaxxing-url.txt, evidence/lmx/matrix-20261111T173228Z.p150.json; table in Speed).

Precision as served

1.459 GB of weights are streamed per decoded token; checkpoint/manifest.json (streamed_weight_bytes_per_token), NOTES-quant.md. Everything is resident in the P150's 32 GB DRAM. ckpt-bfp8 below is an all-BFP8 round-to-nearest variant (1.836 GB per token) that was built and measured for comparison; it is not part of this release.

Quality

Teacher-forced against the original BF16 model on CUDA (RTX 3090, Transformers 5.21.0, eager attention, FP32 logits) on 38 records / 57,876 scored tokens (chat 16,758, instruct 1,216, code 23,522, book 16,380).

TT P150 (this checkpoint) vs BF16

evidence/logs/prefill2/eval_q_r2.json (device run of the dev-s10 runtime, evidence/logs/prefill2/eval_q_r2, 2048-token chunked prefill; the dev-s9 runtime: evidence/quant/fq/tt-ckpt.score.json, all +0.31% / top-1 0.9316 / KL 0.0244)

Chat PPL drops under this quantization (as in most BFP4 rows of NOTES-quant.md) while the other corpora rise, so the all ΔPPL partly cancels; KL (which cannot cancel) is the primary measure. Instruct has only 1,216 tokens (dNLL standard error ±0.010): its ΔPPL is within noise. The dev-s10 prefill (more exact attention and MLP arithmetic, NOTES-runtime.md section 15) moves the device ΔPPL from +0.31% to +0.55%, next to the fake-quant of the same weights on CUDA (+0.50%), with KL 0.0244 → 0.0243.

Noise floor and quantization ablations

Fake-quant rows run the HF model on the 3090 with the Linear weights replaced by the exact lfmq code values (evidence/quant/fq/<row>.score.json); the noise floor is the BF16 model itself with 512-token chunked prefill vs one pass (evidence/ref/eval/noise-chunk512.json).

GPTQ halves the KL of round-to-nearest BFP4 MLPs (0.0689 → 0.0387); keeping w2 at BFP8 brings it to 0.0239 for 0.19 GB more per token, the cheapest tested allocation meeting KL ≤ 0.03 and |ΔPPL| ≤ 1% (NOTES-quant.md). The device matches the fake-quant of the same weights (KL 0.0243 vs 0.0239). ckpt-bfp8 (not released) is the high-fidelity option (KL 0.0017 on device) at 25% more bytes per token.

GSM8K (first 50 test questions, thinking on, max 4,096 new tokens)

50 questions is a sanity check, not a benchmark. Greedy outputs differ from BF16 text even when answers agree: long reasoning traces amplify single flipped near-ties; dev-s10's three greedy misses are 4,096-token caps (dev-s9 solved items 8 and 41 in ~2,850 tokens), and the seeded runs score the same as dev-s9 (98/100 each).

Speed

OpenAI API over HTTP on 127.0.0.1, streamed completions, 512 output tokens (ignore_eos), fresh Opticks windows (no prefix-cache hits), median of 3, lfm3/tests/http_bench.py via tools/bench_serving.sh; host load average (1 min) during each table in its caption. "greedy" = temperature 0, repetition_penalty 1.0 (device argmax); "default" = the served default sampling (T 0.1, top-k 50, repetition 1.1, on the exact device sampler). Decode tok/s = (tokens − 1) / (last chunk − first chunk). The decode tables were measured on dev-s9: dev-s10 and dev-s11 change only prefill (decode step and speculation unchanged); their TTFT column is dev-s9's, the TTFT and concurrent tables below are dev-s11 (tools/bench_serving.sh dev-s11 ttft long concurrent). dev-s12 (decode-ahead at any number of users, NOTES-runtime.md section 17) changes multi-user decode: the 16-slot table below; one user on a 1-slot server is unchanged within 1-2% (evidence/logs/bench/dev-s1{1,2}/summary_decode1.txt). dev-s13 (this release: speculation in shared rounds, NOTES-runtime.md section 18) changes multi-user decode with speculation (the workload and matrix tables below); one user on a 1-slot server is unchanged (ship gate, seeded median of 3: chat_2k 335.0, code_edit 1,497.1, tool_write 1,537.7, gsm8k 336.1 tok/s vs dev-s12 334.4 / 1,509.4 / 1,543.5 / 332.3; evidence/logs/serving/dev-s13/spec_table.md, evidence/logs/serving/dev-s12/spec_table.md).

Single user (max-num-seqs 1, max-model-len 131,072)

GPTQ checkpoint, speculation off / on (evidence/logs/bench/dev-s9/decode_b1_q_nospec.json, decode_b1_q_spec.json; load 1.0–2.4). In parentheses: acceptance length (1 + accepted drafts per verify step).

ckpt-bfp8 (decode_b1_bfp8_nospec.json, decode_b1_bfp8_spec.json; load 1.0–2.0):

The "on" columns continue a book excerpt, whose phrasing the model partly repeats, so prompt lookup finds drafts there; the workload table below is the better guide. Speculation never changes the output: on and off give the same text (18/18 seeded, 6/6 greedy; evidence/logs/bench/dev-s9/spec_table*.md, evidence/logs/serving/dev-s9/summary.txt).

Prompt-lookup speculation by workload (1 user, GPTQ)

evidence/logs/bench/dev-s9/spec_table.md (default sampling, median of seeds 1–3; load 1.0–1.2), workloads of lfm3/tests/http_spec_bench.py:

Greedy, one seed (spec_table_greedy.md): 273.8 → 375.4, 269.6 → 1,783.0, 268.8 → 988.4, 269.3 → 1,646.7, 276.5 → 412.8, 275.0 → 287.2 tok/s. No workload is slower with speculation on.

Time to first token (1 user)

GPTQ, evidence/logs/bench/dev-s11/prefill_b1_q.json (max-model-len 131,072, load 2.0–2.4) and prefill_b1_q_262k.json (max-model-len 262,144, stock decode step: see Limitations; load 1.2–1.3); dev-s9 / dev-s10 from evidence/logs/bench/dev-s9/, evidence/logs/bench/dev-s10/ (same files, same method). A 1-token completion, median of 3; prefill = prompt / TTFT.

dev-s11 changes only the attention of later chunks (2,048 differs by noise: one chunk at start 0, host load 2.4). ckpt-bfp8 (prefill_b1_bfp8.json): 0.053 / 0.185 / 0.939 / 6.26 s at 2K / 8K / 32K / 130K (dev-s10: 0.053 / 0.190 / 1.03 / 7.57; dev-s9: 0.079 / 0.302 / 1.61 / 12.3). Exact prefix caching (same files): repeating a prompt resumes at its last 2,048-token boundary: 2,112 tokens 0.076 → 0.024 s, 32,832 tokens 0.888 → 0.036 s, 129,088 tokens 5.96 → 0.081 s; the resumed output equals the cold one (64/64 greedy tokens at 2,112 and 32,832, evidence/logs/serving/dev-s11/http_checks_b1_q.json). One 2,048-token chunk on the device (evidence/logs/prefill3/prof_q.json; dev-s10 evidence/logs/prefill2/prof_final_q.json, dev-s9 in parentheses): 37.3 ms at start 0 (37.9; 59.3), 40.1 at 8,192 (43.6; 75.5), 53.0 at 32,768 (65.9; 118.4), 102.5 at 126,976 (151.6; 282.5); NOTES-runtime.md sections 15 and 16.

Concurrent users, the matrix's 16-slot server (dev-s11 → dev-s12)

One server with max-num-seqs 16, max-model-len 131,072, GPTQ, N simultaneous streams of 512 tokens (tools/bench_serving.sh TAG multi: evidence/logs/bench/dev-s11/multi_b16_q.json → evidence/logs/bench/dev-s12/multi_b16_q.json). Per-user decode tok/s greedy / default; aggregate (all tokens / round wall, incl. the N prefills) greedy in parentheses. dev-s12 launches the next decode step of all running users as soon as a step's tokens are read, so vLLM's per-step host work (0.5–0.8 ms, evidence/logs/hostprof/base_table.md) overlaps the device step.

At 32,768 tokens the rounds are dominated by the N prefills (0.73 s each, interleaved with the other users' decode). Steady decode with every user active (evidence/logs/hostprof/after_table_32k.md, dev-s12): 245 / 431 / 702 / 1,026 / 1,311 tok/s aggregate at 1 / 2 / 4 / 8 / 16 users; at 128 tokens 278 / 551 / 1,062 / 1,989 / 3,484 (after_table_128.md). Concurrent outputs equal each request alone and equal decode-ahead off, greedy, seeded and mixed sampling, with joins, leaves, an abort and stop strings (Verification).

Concurrent users with speculation, 16-slot server (dev-s12 → dev-s13)

dev-s13 runs prompt-lookup speculation for every running user in one shared verify round; dev-s12 (and dev-s11 above) speculates with one user only, so the 16-slot tables above ran without speculation. Workloads of lfm3/tests/http_spec_bench.py (chat_2k: a 2K-token document + an original poem; code_edit / tool_write: rewrite a file / one write_file call with a file; gsm8k: a reasoning question), default sampling, median of 3 (tools/bench_serving.sh TAG workloads; evidence/logs/bench/dev-s12/workloads_b16_q.json → evidence/logs/bench/dev-s13/workloads_b16_q.json, per-cell acceptance and load in evidence/logs/bench/dev-s13/summary_workloads.txt, load 1.7–11.3). Aggregate tok/s (per-user tok/s); the table is the one in "what changed" above:

32,768-token prompts, 512 tokens, per user greedy / default, aggregate greedy incl. the prefills in parentheses (evidence/logs/bench/dev-s1{2,3}/workloads_b16_q_32k.json): 1 user 243.5 / 237.1 (182) → 481.5 / 327.7 (287); 2 users 210.5 / 180.8 (261) → 340.1 / 337.4 (312); 4 users 170.7 / 168.6 (329) → 183.2 / 183.3 (337); 8 users 76.7 / 75.9 (380) → 75.2 / 74.6 (374); 16 users 41.3 / 41.1 (410) → 41.0 / 40.7 (409).

Concurrent users, 32,768 server (dev-s11, speculation off: dev-s11 speculates with one user only)

One server with max-num-seqs 16, max-model-len 32,768, GPTQ; N simultaneous streams of 512 tokens (evidence/logs/bench/dev-s11/concurrent_b16_q.json). Per user = median over the N users; aggregate = all tokens / round wall (includes the N prefills); steady = N × per user. Cells: greedy / default sampling. Last column: dev-s9's TTFT (evidence/logs/bench/dev-s9/concurrent_b16_q.json).

2K prompts are one start-0 chunk each, so concurrent TTFT is dev-s10's within noise (dev-s10: 4 × 2K 0.154 / 0.187 s, 16 × 2K 0.668 / 0.710 s, evidence/logs/bench/dev-s10/concurrent_b16_q.json).

Prompts are prefilled one at a time, so concurrent TTFT grows with N × prompt. Concurrent greedy outputs equal the same requests served alone (64/64 × 4, evidence/logs/serving/dev-s11/http_batch4_q.json); the decode step is batch-invariant (a slot's logits at any width are bitwise its width-1 logits; NOTES-runtime.md section 10).

LocalMaxxing matrix (llm-inference-bench 0.7.7, lmx speed-test matrix run)

Same command shape as the LFM2.5-24B-A2B P150 matrix: one server, LFM3_MAX_NUM_SEQS=16 LFM3_MAX_MODEL_LEN=131072 tools/serve_lfm3.sh (= launch.py --max-num-seqs 16: KV pool 1,572,864 tokens, liquid default sampling, prefix caching on; speculation: off on dev-s12, which speculated on 1-slot servers only; on in shared rounds at every concurrency on dev-s13); lmx speed-test matrix run --base-url http://127.0.0.1:8000 --hf-id LiquidAI/LFM3-1.7B --quantization BFP4 --concurrency 1,2,4,8,16 --contexts 0,4k,16k,32k,64k,120k --prefill-contexts 2k,8k,32k,64k,128k (30 s per cell, ignore_eos, max_tokens 8,192; the bench sends no sampling fields: unseeded liquid sampling on the exact device sampler). Aggregate decode tok/s (per-user tok/s), dev-s12 (release s12) → dev-s13 (this release):

dev-s13: 1 user +13–23% at context 0–64K, 2 users +9–13% at 0–64K (shared rounds; the bench generates free text, where prompt lookup gains like the chat workloads above, not like the copy workloads); at 120K and with 4–16 users within −2.4% .. +2.9% (plain rounds when no draft pays or no row is free). No cell was flagged by the bench's loop check. Cold prefill (scout request, TTFT / tok/s) is unchanged: 2K 0.074 s / 29,265; 8K 0.189 s / 43,482; 32K 0.731 s / 44,256; 64K 1.82 s / 35,414; 120K 4.86 s / 24,858. Evidence: evidence/lmx/matrix-20261111T173228Z{.bench,.p150}.json, dry-run evidence/lmx/matrix-20261111T173228Z.p150.dryrun.txt (valid, 37 cells, Tenstorrent P150; run script evidence/logs/lmx/run_s13b.sh, run log evidence/logs/lmx/lfm3-run-s13.err, server evidence/logs/lmx/serve16_s13.log, import evidence/logs/lmx/import-s13.txt). Published on LocalMaxxing: https://www.localmaxxing.com/en/inference-benchmarks/cmwcf0qxy01gdmi01twvma3jz (evidence/lmx/localmaxxing-url.txt). Two earlier dev-s13 runs of the same command (evidence/lmx/matrix-20261111T154624Z*, matrix-20261111T171129Z*, both dry-run valid) each had one cell flagged by the loop check (16K × 2, 16K × 1: an unseeded stream that cycled), the same kind of one-off sampling draw as the first dev-s12 run; every seeded / greedy gate is token-identical with speculation off.

dev-s11 → dev-s12 (decode-ahead at any number of users): +10–14% at context 0–4K, +5.5–12% at 16K–32K, +3–11% at 64K–120K (evidence/lmx/matrix-20261030T114534Z* → evidence/lmx/matrix-20261030T192001Z*, run evidence/logs/lmx/run_s12.sh, lfm3-run-s12.err, serve16_s12.log; the first dev-s12 run evidence/lmx/matrix-20261030T185754Z* had one flagged cell). The dev-s12 matrix is published on LocalMaxxing for release s12: https://www.localmaxxing.com/en/inference-benchmarks/cmvve25b60pwgs601c0ixry9g.

Device step (traced decode step, no HTTP)

evidence/logs/fast_step_q_b{1,2,4,8,16}.json, evidence/logs/fast_step_bfp8_b*.json (NOTES-fast-kernels.md section 0; every row active at the position; load 0.7–2.1). ms per step / aggregate tok/s:

Weight roofline at 470 GB/s: GPTQ 3.10 ms, ckpt-bfp8 3.91 ms per step; one row runs at 1.08–1.12× of it. The HTTP server adds ~0.2 ms per token (3.66 ms period with decode-ahead, NOTES-runtime.md section 12); dev-s12: device step

• 0.1–0.15 ms at 1–16 users (NOTES-runtime.md section 17).

From stock to fused to speculation (1 user, GPTQ, 2K prompt, greedy over HTTP)

Prefill, same server (TTFT, median of 3; evidence/logs/bench/dev-s{9,10,11}/prefill_b1_q.json):

Fused kernels

The decode step is one traced sequence of 74 programs (stock: ~1,000), 3.48 ms on device at position 2,000 (NOTES-fast-kernels.md; device profile evidence/logs/prof_step_q.json):

• FusedMLPDecode: sandwich residual + FFN RMSNorm + w1|w3 SwiGLU + w2 in one program on 64 cores, streaming the BFP4/BFP8 weights at 478–485 GB/s (72.5 µs per layer, at the weight roofline).
• ConvOpDecode: sandwich residual + operator norm + in_proj + B·x + 7-tap causal conv with the 6-row state shifted in-kernel + C· + out_proj, one program per conv layer (13 layers).
• AttnNormQKVFront → SDPADecode → o_proj stream: norm + fused q|k|v|gate projection + per-head q/k RMSNorm + RoPE (zero-tail partial rotary on full layers) + paged BFP8 KV write; paged SDPA decode (head_dim 128, GQA 4; sliding layers read only their ≤ 5 window blocks) with the sigmoid head gate in an FP32 epilogue.
• DRAM-streaming decode matmuls from bank-contiguous weight copies (BFP8 / BFP4 / BF16) and the 128K-vocab LM head at 490 GB/s; a final residual + norm in one launch.
• Prompt-lookup speculation: verify steps of 2 / 4 / 8 / 16 rows through the same fused modules (conv rows chained over the state, one KV-tile leader per 32 rows), device argmax or exact sampling of every row, and SpecCommit (accepted rows' conv state); bitwise the sequential steps (evidence/logs/serving/dev-s9/test_speculate.json). Since dev-s13 a verify step is shared by all running users: a row map gives each row its offset in its user's group, the group length and the user's conv-state slot; conv verify chains each group from its own slot, and SpecCommit commits each group's slot and sampler history (NOTES-fast-kernels.md section 7).
• Decode-ahead (any number of users since dev-s12): the running users' next step is launched as soon as a step's tokens are read, overlapping vLLM's host work; the scheduler allocates one lookahead KV slot per request and runs a pending step's decode turn before admitting a prefill; finished rows are condensed on the device.
• Prefill (dev-s10, NOTES-runtime.md sections 11 and 15): per layer, a fused attention front (q heads, q/k RMSNorm, RoPE, paged BFP8 K/V write: 12 programs → 1), a paged GQA prefill attention kernel (each K/V tile read once for the 4 q heads of its group, causal triangle split evenly over all 110 cores, sliding layers read only their 256-key window, FP32 softmax accumulation, sigmoid gate + head concat in the epilogue), a w1|w3 SwiGLU kernel (silu·mul on the FP32 accumulators, no g / u round trip through DRAM), a bitwise post-norm + residual + next-norm kernel on all cores, and the 7-tap conv prefill kernel on all cores; 187 programs per 2,048-token chunk (dev-s9 549–599). dev-s11 (NOTES-runtime.md section 16): the attention kernel's exp is an SFPLOADMACRO pipeline (5 issued SFPU instructions per row, was 16) and full layers stack 4 query row tiles per unit for chunks ≥ 1,024 rows (2 below); full-layer attention at 126,976 20.9 → 11.9 ms per layer, chunk at start 0 59.3 → 37.3 ms, at 126,976 282.5 → 151.6 (dev-s10) → 102.5 ms device time.
• Exact on-device sampling of the BF16 logits (vLLM temperature / top-k / top-p / min-p / penalties / seed semantics, equal to a host replica), device argmax for greedy.

Sampling, chat template, parsers

• Thinking. The chat template's generation prompt ends with <think>: the model reasons first, and the lfm3 reasoning parser returns the text before </think> as message.reasoning and the rest as content (also when a tool call ends the reasoning without </think>). Per request, "chat_template_kwargs": {"enable_thinking": false} puts <think></think> in the prompt and the whole output is content. Server default: --thinking on (default) or --thinking off (off unless a request turns it on; used for agent harnesses).
• Tool calls. The model emits Qwen3-Coder-style XML: <tool_call>\n<function=NAME>\n<parameter=K>\nV\n</parameter> \n</function>\n</tool_call> (all six tags single tokens). The lfm3 tool parser returns OpenAI tool_calls (JSON arguments typed by the request's schema; strings verbatim, multi-line included; finish_reason: tool_calls). Streamed responses equal non-streamed ones for any delta size (see Verification): a call arrives as its header (id, name) followed by its complete arguments once the call is closed, plus empty-argument keep-alive entries while it is being generated.
• Default sampling comes from generation_config.json: temperature 0.1, top-k 50, repetition penalty 1.1 (the liquid preset; --sampling-preset math (T 1.0, top-p 0.95, top-k 20), agentic (T 0.6, top-p 0.95), greedy; tools/serve_lfm3.sh also takes arbitrary LFM3_DEFAULT_SAMPLING JSON). A request's own fields override the default one by one. Temperature / top-k (incl. disabled) / top-p / min-p / presence / frequency / repetition penalties / seed run on the device sampler; logprobs, logit_bias, allowed_token_ids and structured outputs fall back to vLLM's host sampler. For bit-exact greedy pass temperature: 0 and repetition_penalty: 1.0 (the default keeps repetition 1.1).

Requirements

• One Tenstorrent Blackhole P150 (32 GB), tt-kmd driver, /dev/tenstorrent, 1 GiB hugepages (/dev/hugepages-1G) as for any tt-metal workload; the card must be idle and owned by you.
• Docker, Python ≥ 3.11 (standard library only), ~30 GB disk for the download (2.0 GB checkpoint + 5.5 GB archive, plus 22.1 GB once the image is loaded), ~3.9 GB for the converted-tensor cache, ~1 GB of kernel cache.
• Device DRAM: ~2.4 GB of weights + 1.4–1.8 GB of decode streaming copies + 0.41 GB of prefill w1|w3 panel copies (NOTES-runtime.md sections 2 and 15, NOTES-fast-kernels.md section 1), KV 11.7 KB/token (one 131,072-token user: 1.5 GB).
• Start with warm caches: ~30 s from launch to "Application startup complete" (evidence/logs/serving/dev-s11/serve_b1_q.log); the first start with empty caches converts the weights and compiles kernels (see Verification for the measured time).

Download and serve

curl -LO https://huggingface.co/Lottolabs/LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150/resolve/main/launch.py

# Download and verify only (no Docker or device needed)
python3 launch.py --download-only --cache-root /large-disk/lfm3

# One user, 131,072-token context, prompt-lookup speculation on, thinking on, prefix caching on (defaults)
python3 launch.py --cache-root /large-disk/lfm3 --device-ownership-confirmed

# 16 concurrent users, 131,072 tokens each (also 2 / 4 / 8; speculation in shared rounds for all users)
python3 launch.py --cache-root /large-disk/lfm3 --max-num-seqs 16 --device-ownership-confirmed

# Agent harness in other containers: thinking off by default, reachable on the docker bridge
python3 launch.py --cache-root /large-disk/lfm3 --thinking off --host 172.17.0.1 --max-model-len 65536 \
    --device-ownership-confirmed

# Print the exact docker command; options: --max-model-len N, --speculate off, --sampling-preset
# liquid|math|agentic|greedy, --prefix-caching off, --port P, --revision REV
python3 launch.py --cache-root /large-disk/lfm3 --print-command

The KV pool is max-model-len × max-num-seqs, capped at 1,572,864 tokens (18.4 GB). max-model-len is at most 131,072 (the fused decode step's limit; see Limitations). --release-dir DIR serves an already downloaded release directory in place (still verified byte for byte). The server listens on http://127.0.0.1:8000/v1 (--host to publish on another address), model id LiquidAI/LFM3-1.7B:

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "LiquidAI/LFM3-1.7B", "max_tokens": 1024,
  "messages": [{"role": "user", "content": "What is 17 * 23?"}]}'

Stop with Ctrl-C or docker stop lfm3-1.7b-api. The launcher's Docker command is argument-for-argument the reference tools/serve_lfm3.sh that produced the measurements on this card (its environment knobs map to the launcher's flags: LFM3_MAX_NUM_SEQS, LFM3_MAX_MODEL_LEN, LFM3_SPECULATE, LFM3_THINKING, LFM3_SAMPLING_PRESET, LFM3_PREFIX_CACHING, LFM3_HOST, LFM3_PORT). The launcher never stops containers, resets the card or takes the device from other processes.

Cache safety. The tensor cache is bound to the checkpoint-manifest SHA256 and the image ID through lfm3-cache-identity.json; a cache built for anything else, or a non-empty directory without the identity file, is refused.

Verification

Gates passed by the release image (= dev-s13; tools/ship_serving.sh dev-s13, evidence/logs/serving/dev-s13/summary.txt; every dev-s12 gate re-run, file names below are in that directory):

• Speculation in shared rounds (dev-s13, NOTES-runtime.md section 18): model level, eager shared verify rounds bitwise == sequential == alone (15/15 rounds); decode-trace switches (the page-table regression: every decode trace stepped with every other one replayed in between == re-staged, cached inputs == device) 10/10; eight requests of mixed greedy / liquid / agentic / reasoning sampling with joins, leaves, an abort and remaps, with forced drafts and with the model's own plans, decode-ahead off and on: every request == itself alone (test_speculate*.json). 16-slot server at 2 / 4 / 8 / 16 users, new groups (copy prompts whose drafts land = shared verify rounds, tool-call chats streamed and non-streamed, copy abort, copy stop) plus the dev-s12 groups: every reply == alone, and identical with LFM3_DECODE_AHEAD=0 and with LFM3_SPECULATE=0 (362/362 each; http_concurrent_b16_{ahead,noahead,nospec}.json).
• Decode-ahead with several users (since dev-s12): model level, 4 slots, six greedy / seeded requests with staggered arrivals, condensing and an abort: every request == itself alone with decode-ahead off, with the scheduler handshake, without lookahead slots, and with admissions while a step is pending; on-device row condensing == the host permutation bitwise (test_decode_ahead.json).
• Model level: traced decode == eager (32/32 greedy tokens, 3.56 ms step wall), exact device sampler == host replica (8/8 × 2 presets) (adapter.txt).
• HTTP, 1 slot, GPTQ: "17 × 23" → reasoning split off, content 391; default-sampling chat; get_weather and multi-line write_file tool calls, streamed == non-streamed; thinking off; HTTP greedy == model-level greedy 64/64; prefix resume == cold 64/64 at 2,112 / 32,832; GSM8K first 10: 10/10; sampling matrix (44 cases) and seeds PASS (http_checks_b1_q.json, gsm8k10_b1_q.jsonl, http_sampling_*_b1_q.json).
• Speculation on == off (1 slot): same text 18/18 seeded + 6/6 greedy (spec_table*.md).
• Streaming with speculation on (since dev-s9): 15 requests streamed and non-streamed with the same seed / greedy (same tokens 15/15): the 179-line write_file rewrite (4 runs, one cut by max_tokens inside the call), the same with thinking on, the two CRUDbench Pi requests whose streamed calls lost their arguments before, a two-call request: content, reasoning, tool calls and finish_reason identical 15/15 (http_stream_tools.json). Host replay of those outputs and 14 synthetic ones through the image's parsers and a copy of vLLM's streaming path, cut 1 / 2 / 3 / 4 / 5 / 7 / 8 / 11 / 16 / 32 tokens at a time, whole, 24 random cuts and the server's own cut: streamed == non-streamed for every case without a known deviation (980 of 1,011 splits; the rest are the documented cut inside a parameter name, see Limitations); vLLM's original parser on the same cuts 305 of 1,011 (check_tool_stream.json).
• 4 slots: concurrent == alone 64/64 × 4, checks pass (http_batch4_q.json, http_checks_b4_q.json); ckpt-bfp8 (not released): checks pass, HTTP greedy == model-level 64/64 (http_checks_b1_bfp8.json).
• Prefill (dev-s10, NOTES-runtime.md section 15): every new kernel tested against the dev-s9 op chain and an FP64 reference (evidence/logs/prefill2/): K/V cache writes, conv output / state and the residual norms bitwise; attention error vs FP64 6–40× lower; MLP error 3–7% lower; module tests and BF16-golden greedy match lengths unchanged (37 / 64 / 17 / 24 with the stock decode step), eval KL 0.0243 (dev-s9 0.0244).
• Prefill attention (dev-s11, NOTES-runtime.md section 16, evidence/logs/prefill3/): real attention inputs at chunk starts 0 / 8,192 / 32,768 / 126,976, 2,048- and 128-row chunks: error vs FP64 0.0017–0.0019 (gated ≤ 0.0025, = dev-s10), permuted page table bitwise (sg_real_adaptive.json); goldens 37 / 64 / 17 / 24 with the stock and the fast decode step (t_model_q.json, t_model_q_fast.json); eval KL 0.0243 (eval_q.json).
• Earlier model-level gates (NOTES-runtime.md sections 7, 10): every layer vs the HF reference (output PCC ≥ 0.9998 per layer on golden inputs), full model vs BF16 goldens (greedy divergences only at near-ties where the CPU BF16 run also diverges), checkpoint verify (287 tensors re-checked against the original, all file hashes; checkpoint/verify.json).

Release check (2026-11-12, evidence/release-verification/local-staging-serve.json, load 3.3–5.2): launch.py verified all files of the staged release (245 files, 7,582,701,843 bytes, before this card was added to the inventory); its Docker command is argument-for-argument the reference tools/serve_lfm3.sh in 5 configurations (defaults; 4 sequences; 16 sequences; 16 sequences + thinking off + host 172.17.0.1 + 65,536 + agentic preset; speculation / prefix caching off + greedy preset + port; the speculative config is passed at 1, 4 and 16 sequences), and the server it started from empty caches (2 min 50 s from launch to ready, including the checksum pass; 3.9 GB tensor cache, 1.2 GB kernel cache) answered: greedy chat "391" (17 × 23) with the reasoning split off; a default-sampling chat; a tool call get_weather({"city": "Paris", "unit": "celsius"}) with finish_reason: tool_calls, streamed == non-streamed; GSM8K first 10: 10/10; 2K-prompt greedy book continuation 628 tok/s (speculation on, median of 3), 2K chat essay 320 tok/s, TTFT 0.044–0.050 s. A second server with --max-num-seqs 4 (same caches, 1 min 20 s to ready) served four different requests at once (a copy-heavy code edit, a GSM8K question, a greedy chat, a seeded default-sampling chat) with the same text, finish reason and token count as each alone (4/4); vLLM counted 192 draft proposals (1,318 draft tokens, 1,011 accepted) during the concurrent phase.

Published-file check: in the follow-up commit (anonymous download of the release commit + serve).

CRUDbench (agentic coding, 40 tasks)

CRUDbench 0.4.1, concurrency 1, thinking off, speculation on, image dev-s8 (same runtime as dev-s9); NOTES-crudbench.md, evidence/crud/results/*.json. These CRUDbench runs were not submitted to localmaxxing.com.

Pi ties LFM2.5-2.6B (13/40); the native harness scores half of it (the model rarely ends with a final message: 37/40 tasks hit the turn cap, and with thinking off it sometimes writes its reasoning into the write_file content). Pi runs loop on the same failing test until the 65,536-token context overflows (vLLM 400; 16 tasks in B1). Prompt lookup accepts 51–73% of drafts in these runs (prefix-cache hit 79–95%).

The streaming bug and its fix. Pi streams. With speculation, a whole tool call arrived in 1–3 SSE deltas, and vLLM's incremental Qwen3-Coder parser then sent the call name with arguments {} (492 of 523 streamed calls in one smoke task), so Pi looped on "Validation failed … must have required properties command". The runs above used a proxy that requests non-streamed responses and replays them as SSE. dev-s9 fixes it in the server: LFM3's own streaming tool parser emits each call's complete arguments once the call closes, for any delta size, and a vLLM patch keeps the reasoning text of the delta that ends the reasoning (NOTES-runtime.md section 13). Re-run of the broken smoke task without the proxy: 79 streamed tool calls, 0 with empty arguments, 0 validation failures (evidence/crud/results/smokeB4-pi-direct-s9-toolargs.json; before: 14 / 16 of 141 direct, 492 / 492 of 522 via the log proxy); the task itself still failed (0/1) by looping into the context limit.

Limitations

• Text only. Single P150 (no multi-card). With the fused decode step max-model-len is ≤ 131,072: at 262,144 its SDPA program needs 1,304,064 B of L1 per core (limit 1,300,000) and the server does not start (evidence/logs/bench/dev-s9/serve_b1_q_spec_262144_fastdecode_fail.log); launch.py refuses it. The runtime runs up to 262,144 tokens with the stock decode step (LFM_FAST_DECODE=0 LFM3_SPECULATE=0 in tools/serve_lfm3.sh from the runtime archive, ~63 tok/s decode; prefill speed unchanged), which launch.py does not offer.
• Prompt-lookup speculation helps where the output repeats the context (edits, rewrites, tool calls with file contents: up to 5.9× with one user, 3.3× per user with two) and little on novel text (1.04–1.4×); with 16 busy users there are no free rows for drafts and it runs plain steps (same speed as without).
• Quantized: expect small differences vs BF16 (KL 0.024, top-1 93%). The all-BFP8 variant (KL 0.002, ~17% lower decode speed) is not part of this release; quant/build_ckpt.py in the runtime archive builds it from the original.
• Streamed tool-call arguments arrive per call (complete JSON when the call closes), not token by token. A response cut by max_tokens inside a parameter name streams the call with the parameters complete so far, whereas vLLM's non-streaming parser returns that raw text as content (lfm3/tests/check_tool_stream.py, known deviation).
• Prefill is not chunked across requests (one prompt at a time; a long prompt delays other users' first tokens); prefix caching resumes at 2,048-token boundaries.
• As an agent (CRUDbench) the 1.7B model loops on failing tests and, with thinking off, can write reasoning into tool arguments.
• Experimental software on a shared research host; numbers are from one P150 on 2026-10-27 – 2026-11-12 and vary with host load.

License and attribution

The model weights are a derivative of LFM3-1.7B by Liquid AI, licensed under the LFM Open License v1.0 (full text: LICENSE, also in checkpoint/LICENSE). Liquid AI's original model card is retained as NOTICE-original-model-card.md.

• Commercial use limit (Section 5): commercial use is licensed only for entities with annual revenue below US $10,000,000 (the "Threshold"); commercial use by an entity at or above the Threshold is not licensed (the Threshold does not apply to qualified non-profits for non-commercial or research purposes). Read the License before using this release commercially.
• Modifications notice (Section 4(b)): Lottolabs modified the model weights: MLP w1/w3 were quantized with GPTQ to Tenstorrent BFP4 and w2 with GPTQ to BFP8, attention / conv / LM-head weights were rounded to BFP8, and all tensors were re-packed into the lfmq-1 safetensors layout (checkpoint/layer-*.safetensors, checkpoint/global.safetensors, checkpoint/manifest.json). config.json, generation_config.json, configuration_lfm3.py, tokenizer files and the chat template are unmodified copies.
• The runtime image bundles tt-metal, vLLM and the vLLM TT plugin under their own licenses (inside the image); the LFM3 runtime overlay source is in runtime/lfm3-runtime-src.tar.gz, with our vLLM / plugin patches in docker/.
• "LFM" and "Liquid" are Liquid AI's names, used here only to describe the origin of the work.

Provenance

• Base model: LiquidAI/LFM3-1.7B, revision 40ebe0aa6662c9d40389c2f5d565b6a2c6aebacd (checkpoint/manifest.json).
• GPTQ calibration (quant/build_calib.py in the runtime archive): 128 × 2,048 tokens + 8 held-out, seed 1: 48 UltraChat train_sft chats, 32 Python source files (site-packages modules), 16 Dolly + 16 Alpaca instruction chats, 16 Project Gutenberg prose chunks; rendered with the LFM3 chat template. Checked disjoint from the evaluation texts, goldens and GSM8K prompts (no shared 32-token window; evidence/quant/data/overlap.json).
• GPTQ (quant/gptq_lfm3.py, RTX 3090): sequential over layers (each calibrated on the outputs of the already quantized layers), BFP block quantizer in TT geometry, act-order, block 128, damping 0.01 × mean diag(H), clip search 2; w1/w3 on the FFN-norm inputs, w2 on the quantized-w1/w3 activations; settings, calibration summary and per-layer metric hashes in checkpoint/manifest.json. Verified by quant/verify_ckpt.py (checkpoint/verify.json).
• Evaluation texts: chat (UltraChat test), instruct (code/multilingual), code (Python files) and book (Opticks) records, re-rendered with LFM3's template; GSM8K: first 50 questions of the test split (evidence/ref/gsm8k/gsm8k-prompts.jsonl).
• Serving image lottolabs/lfm3-1.7b-tt-p150:s13-20261112 (= build dev-s13, 2026-11-11) FROM a tt-metal + vLLM (TT plugin) P150 image with the overlay in runtime/lfm3-runtime-src.tar.gz (docker/Dockerfile).