🇯🇵 LFM2.5-1.2B-JP-202606-NVFP4
NVFP4 (W4A4) quantization of LiquidAI/LFM2.5-1.2B-JP-202606 — Liquid AI's latest general-purpose Japanese chat model (1.2 B-param dense lfm2 hybrid: short-convolution + attention, 128 K context). The 4-bit build is 1.1 GiB on the GPU and leaves room for a ~1.87 M-token KV cache on a single 16 GB Blackwell card.
Quantized by Lna-Lab with NVIDIA TensorRT Model-Optimizer (modelopt). Loads directly in vLLM as modelopt_fp4 — no checkpoint surgery, no --quantization flag required (auto-detected).
NVFP4 (W4A4) 量子化版 — リキッドAIの最新汎用日本語チャットモデル（1.2B dense lfm2、短畳み込み＋注意のハイブリッド、128Kコンテキスト）を4bit化。GPU上の重みは1.1 GiB、16GBのBlackwell 1枚で約187万トークンのKVキャッシュを残せます。
Why it's nice: the model is tiny (~1.2 B dense) and edge-tuned, so 4-bit weights turn the whole 16 GB card into KV + concurrency headroom. On a single power-capped RTX PRO 2000 it sustains 24 k tok/s aggregate; two cards run as independent replicas for 46 k tok/s — at ~330 tokens per joule.
📊 Measured throughput — 1× / 2× RTX PRO 2000 Blackwell (16 GB, SM120, 70 W cap)
vLLM 0.22.0, --quantization modelopt, --kv-cache-dtype fp8, --max-model-len 16384, 256-token pinned decode (ignore_eos+min_tokens), FlashInfer CUTLASS NVFP4 GEMM.
| config | in-flight | aggregate tok/s | per-stream | tok/joule |
|---|---|---|---|---|
| 1 GPU · single stream | 1 | 212 | 212 | — |
| 1 GPU · C256 (knee) | 256 | ≈ 24,200 | ~95 | ~346 |
| 2 GPU · 2× TP1 replicas (DP) | 512 | 🥇 46,020 | ~90 | ~329 |
- Both cards sit at the 70 W power cap / 100 % util at the knee — the throughput is power-bound, not memory-bound.
- For max aggregate throughput on a no-NVLink box, data-parallel (N independent TP1 replicas) beats tensor-parallel — TP would pay a PCIe all-reduce tax every decode step. The 2-GPU number above is simply two single-GPU servers driven together; it scales near-linearly (≈ 2×).
Tensor-parallel scaling (-c 20480 / --max-model-len 20480, fp8 KV, out=256)
| TP | GPUs | board W | max concurrency @20K ctx | KV pool | single stream (C1) | aggregate peak | tok/joule (agg) |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 70 | 91× | 1.87 M tok | 213 | 24,287 (C256) | ~347 |
| 2 | 2 | 140 | 185× | 3.78 M tok | 352 | ~24,200 (C384) | ~173 |
| 4 | 4 | 280 | 409× | 8.39 M tok | 518 | 25,387 (C256) | ~91 |
Aggregate curves (tok/s): TP1 → C64 11,435 / C128 17,886 / C256 24,287 · TP2 → C64 12,798 / C128 18,510 / C384 24,241 · TP4 → C64 13,531 / C128 22,056 / C256 25,387.
Read it as: tensor-parallel scales single-stream latency (213 → 352 → 518 tok/s) and KV concurrency (91× → 185× → 409×, the model shards so each card adds its 16 GB to the KV pool) — but the aggregate plateaus at ~24–25 k tok/s regardless of TP, because one power-capped (70 W) card already saturates there and TP just spreads the same work and adds a PCIe all-reduce tax (no NVLink). So TP buys latency + KV headroom, not throughput. For raw throughput run data-parallel instead (N independent TP1 replicas): 2 replicas → 46,020 tok/s @ 140 W (329 tok/J), ~2× what TP=2 delivers at the same power. Rule of thumb: TP for latency/long-context concurrency, DP for max aggregate throughput & efficiency. TP>1 on a no-NVLink box needs NCCL_P2P_DISABLE=1 NCCL_CUMEM_ENABLE=0 … --disable-custom-all-reduce.
Memory footprint (1 GPU, --gpu-memory-utilization 0.9)
| weights on GPU | 1.13 GiB |
| KV cache (fp8, 16 K ctx) | 10.69 GiB → 1,865,284 tokens |
| max concurrency @ 16 K ctx | ≈ 114× |
Greedy spot-check (temp 0): "The capital of France is" → "Paris."; 「機械学習における量子化とは？」 → 「…精度を低減し数値の範囲を圧縮して計算効率を向上させる手法です。浮動小数点の値を低ビット幅の整数に変換することで、モデルの軽量化と推論速度の高速化を実現します。」 — fluent, accurate Japanese; code & instruction-following preserved.
🔧 What's quantized
A conservative, dense-lfm2-aware recipe — only the SwiGLU MLP is taken to 4-bit:
| kept in BF16 | taken to NVFP4 (W4A4) |
|---|---|
embed_tokens (tied lm_head), all RMSNorms |
feed_forward.w1 / w2 / w3 — every one of the 16 layers |
self_attn.{q,k,v,out}_proj (6 attention layers) |
|
conv.{in_proj, out_proj} + conv.conv (10 short-conv layers) |
- NVFP4 = e2m1 weights, 16-element micro-blocks with e4m3 block-scales + fp32 per-tensor global scale, plus static fp32 input scales (W4A4).
- vLLM fuses
w1+w3 → w13internally (MergedColumnParallelLinear) and merges the per-shard NVFP4 scales — the stock checkpoint key names are used as-is, no remap needed (unlike the MoE sibling). - The depthwise
conv.conv(Conv1d, K=3) is explicitly excluded — block-scaling a 3-tap kernel is meaningless and the weights are negligible. - Calibrated in fp32 + eager on a small JA/EN/code mix (256 samples).
Note — use this W4A4 build, not weight-only. A W4A16 (weight-only) NVFP4 export of this architecture is not servable by vLLM: modelopt still writes a static-FP4 input spec into the config but omits the
input_scaletensors, so vLLM reads a garbage activation scale and emits empty output. This W4A4 build carries the input scales and runs correctly.
🚀 Usage (vLLM)
# Single GPU — serve
vllm serve LFM2.5-1.2B-JP-202606-NVFP4 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--max-model-len 16384 \
--max-num-seqs 512 \
--gpu-memory-utilization 0.90 \
--port 8000
from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="x")
r = c.chat.completions.create(
model="LFM2.5-1.2B-JP-202606-NVFP4",
messages=[{"role": "user", "content": "富士山について3文で教えて。"}],
temperature=0.3, max_tokens=200,
)
print(r.choices[0].message.content)
Max aggregate throughput — run one replica per GPU and load-balance across them
(data-parallel), e.g. CUDA_VISIBLE_DEVICES=0 … --port 8000 and CUDA_VISIBLE_DEVICES=1 … --port 8001.
Needs an SM ≥ 75 (Blackwell here) for the NVFP4 GEMM kernels.
📜 License & credits
- Base model: LiquidAI/LFM2.5-1.2B-JP-202606 — © Liquid AI, LFM Open License v1.0 (
LICENSE). All credit for the model itself goes to Liquid AI; this repo only changes the numeric precision. - Quantization: Lna-Lab, via NVIDIA
modelopt0.44.0. arXiv: 2511.23404. - Use of this model is subject to the base model's license. Quantization may introduce minor quality changes versus the bf16 original.