LFM3-1.7B NVFP4 GGUF (Schur Replay)

Modified weights notice: this repo contains a quantized (NVFP4) derivative of LiquidAI/LFM3-1.7B. Every weight matrix was re-quantized by BennyDaBall: the feed-forward weights in NVFP4, attention, short-conv and the tied embedding/head in Q8_0. Only the F32 norms, conv taps and rope table are Liquid AI's original bytes. Not affiliated with or endorsed by Liquid AI.

NVFP4 GGUF of LFM3-1.7B for llama.cpp and LM Studio: 3.34x closer to BF16 than Liquid's own NVFP4 GGUF as users get it (2.11x when that file is forced to W4A8), 2.27x closer than Liquid's Q4_K_M, 31% smaller and 19% faster than Q8_0, and it runs on stock llama.cpp and LM Studio today. Companion of BennyDaBall/LFM3-1.7B-NVFP4 (ModelOpt safetensors for vLLM / SGLang): same recipe, solved here for llama.cpp's 8-bit activation path.

File: LFM3-1.7B-SR-NVFP4.gguf (1.28 GB). NVFP4 on the feed-forward weights (Liquid's own NVFP4 layer map), my own GPTQ Q8_0 on everything else, tagged W4A8 (general.tensor_extra.prec_a4 = false: 8-bit activations, better quality than llama.cpp's default W4A4 prompt path).

Weights updated 2026-11-04. Two changes, both from my Schur Replay solver. The NVFP4 feed-forward weights were re-solved with a post-pass: coordinate descent on the codes against the GPTQ objective, then an end-state re-pick of the E4M3 block scales, run twice. And every other weight matrix (attention q/k/v/gate/output, short-conv in/out, the tied embedding/head) is now my own Q8_0: GPTQ with a replay-scored block-scale search plus the same post-pass on the int8 grid, instead of Liquid's round-to-nearest Q8_0. Same file size, same speed. KLD to BF16 went from 0.0436 to 0.0428, p99 from 0.605 to 0.570. Liquid's own NVFP4 GGUF (published 2026-11-03) measures 0.1427 on the same build and eval, so this file sits 70% closer to the BF16 model than the vendor file at the same size and speed.

KLD vs the BF16 GGUF (lower is better)

Liquid's official BF16 GGUF as the base, 167,320 tokens of held-out tool calls, chat, reasoning and raw text, ctx 2048, every row measured 2026-11-04 on stock llama.cpp b12110 (CUDA 13.4, RTX 5090):

Speed, same sitting, RTX 5090, stock b12110 (llama-bench tg128 / pp512, 5 runs): this repo 590 / 38.7k tok/s, Liquid's NVFP4 GGUF 597 / 49.5k, Liquid Q8_0 497 / 48.5k, Liquid Q4_K_M 634 / 47.0k. Decode (batch 1) is W4A8 for every NVFP4 file whatever the tag; Liquid's untagged file is faster on prompts because Blackwell's default sends them through the W4A4 path, which is where its KLD comes from.

Ablation on the same build: this file with Liquid's Q8_0 for the non-FF tensors instead of mine measures 0.0430 (p99 0.590); the 2026-11-01 weights measure 0.0436 (p99 0.605). Most of the Q8_0 tail's remaining error is llama.cpp's Q8_1 activation quantization, not the weights.

Runs on stock llama.cpp and LM Studio, old builds included. llama.cpp before b12088 (and LM Studio's CUDA runtime 2.67.0) skips the NVFP4 per-tensor scale in LFM's feed-forward block, so a straight NVFP4 conversion of LFM3 (Liquid's NVFP4 converted with convert_hf_to_gguf) prints "666666..." there. This GGUF folds every per-tensor scale out exactly: power-of-two shifts of the E4M3 block scales (bit-exact) plus the remainder folded into ffn_norm (gate/up) and the scale-invariant sandwich post_ffw_norm (down). Measured on stock b12015; chat + tool calls verified.

How the weights were made

Schur Replay (arXiv 2609.36654): inside the GPTQ loop, each 16-weight block tries a window of E4M3 scale codes plus the Four Over Six map-to-4 scale (arXiv 2512.02010). Every candidate replays the block's GPTQ updates and is scored by its real GPTQ loss (the Schur complement of the not-yet-quantized columns). Each row keeps its best. After the solver, a post-pass walks the codes column by column (coordinate descent on the same GPTQ quadratic) and re-picks each block's scale with the codes fixed, twice over; every accepted move lowers the loss the solver minimized. The Q8_0 layers get the same treatment on the int8 grid: GPTQ with each 32-weight block trying a set of clipped fp16 scales, each replayed and scored by its GPTQ loss, then the post-pass. Hessians come from 599k calibration tokens (tool calls, chat, reasoning, raw text). The chat, tool-call and reasoning answers are the BF16 model's own generations.

Usage

Recommended sampling (Liquid's card): temperature 0.1, top_k 50, repetition_penalty 1.1 (math/STEM: temperature 1.0, top_p 0.95, top_k 20). Thinking is on by default; tool calls use Liquid's XML format.

llama-server -m LFM3-1.7B-SR-NVFP4.gguf --jinja -ngl 99

Or load it in LM Studio (CUDA runtime 2.67.0 tested).

License

LFM Open License v1.0 (copy in LICENSE, unchanged). Key terms, quoted from it:

"You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications" (Section 4)

"The rights granted under this License for Commercial Use are conditioned upon You or Your Legal Entity not exceeding the Threshold." (Section 5(a)), where the Threshold is "annual revenue of 10 million United States dollars ($10,000,000) or more." (Section 1)

Free for any use, commercial included, while you (and your corporate group) make under $10M a year; above that, commercial use needs a license from Liquid AI.

Credits

• Model: Liquid AI, LFM3-1.7B. NVFP4 layer map from Liquid's own LFM3-1.7B-NVFP4.
• Methods: GPTQ (Frantar et al. 2022), Four Over Six (Cook et al., arXiv 2512.02010), Schur Replay (arXiv 2609.36654); the post-pass follows the coordinate-descent refinement idea of CDQuant and ReQuant (arXiv 2608.07019), re-implemented on the NVFP4 and Q8_0 grids with block-scale re-picking.
• Quantization: BennyDaBall.