Block-wise FP8 weight quantization of sensenova/SenseNova-Loop-9B,
the 9B-parameter looped multimodal decoder (32 physical layers executed as 56 layer passes, two-stream mHC,
hybrid linear/full attention). The checkpoint layout matches the official Qwen FP8 releases
(quant_method: fp8, block-wise weight_scale_inv).
e4m3fn, one BF16 scale per 128x128 block (<name>.weight_scale_inv, shape [ceil(out/128), ceil(in/128)]), per-block absmax scaling. Weight-only, no calibration data.quantization_config: quant_method: fp8, fmt: e4m3, weight_block_size: [128, 128], activation_scheme: dynamic — same schema as Qwen/Qwen3-32B-FP8.q/k/v/o_projin_proj_qkv/z and out_projgate/up/down_projattn.qkv, attn.proj, mlp.linear_fc1/2)embed_tokens, lm_head, all norms (BF16)conv1d, A_log, dt_bias (BF16)in_proj_a / in_proj_b (BF16) — these 32-row projections fuse into in_proj_ba (64 rows) in vLLM; the Marlin block-scale path requires 128-aligned shard boundaries, so they stay BF16 (same practice as official Qwen3-Next-FP8). ignored_layers covers all 56 execution-layer indices.mhc_attn / mhc_ffn and loop_gate (FP32, as in the original)Qwen3-VL-*-FP8BF16-vs-dequantized-FP8 parity (RTX 4090, transformers 5.18.0 / torch 2.11.0+cu130): mean relative L2 weight error 2.64% (max 2.66%); greedy generation token-identical on math (3/3) and code (160/160) prompts, late divergence with equivalent content on Chinese/reasoning prompts (correct final answers); fixed-input logits cosine 0.9996, top-1 agreement; image-description smoke test passed on both.
Native FP8 serving (single RTX 3090 24 GB, patched vLLM PR #64832, Marlin W8A16): model weights 10.46 GiB resident, 9.55 GiB KV cache (160k tokens); math/code/Chinese/reasoning prompts all correct and coherent, matching the BF16 reference semantics.
Upstream vLLM PR #64832 implements the
architecture but rejects quantized loads. The fork branch
LeTechLead/vllm sensenova-loop-fp8
lifts that guard (one-line change plus the checkpoint's ignore list); with this checkpoint it
serves FP8 W8A16 on a single 24 GB Ampere card:
vllm serve letechlead/SenseNova-Loop-9B-FP8 \
--served-model-name sensenova-loop-9b-fp8 \
--gpu-memory-utilization 0.90 --max-model-len 32768 --enforce-eager
On FP8-capable GPUs (sm89+) the same file uses native FP8 kernels. Build the fork from source
(torch 2.14.0+cu130, MAX_JOBS=$(nproc) TORCH_CUDA_ARCH_LIST=8.6 pip install -e . --no-build-isolation).
The custom modeling code loads BF16; dequantize at load time with the verified helper shipped in this repo:
hf download letechlead/SenseNova-Loop-9B-FP8 --local-dir SenseNova-Loop-9B-FP8
python dequant_to_bf16.py --src SenseNova-Loop-9B-FP8 --dst SenseNova-Loop-9B-FP8-bf16
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SenseNova-Loop-9B-FP8-bf16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True,
dtype=torch.bfloat16, device_map={"": "cuda:0"}).eval()
For native BF16 or 4-bit deployment use the upstream releases: sensenova/SenseNova-Loop-9B and the official AWQ variant.
Apache-2.0, inherited from the original model. Original weights (c) SenseNova; quantization by LeTechLead.
model-00001.safetensors 2.0 GBmodel-00003.safetensors 2.0 GBmodel-00004.safetensors 1.0 GBmodel-00008.safetensors 987 MBmodel-00006.safetensors 987 MBmodel-00007.safetensors 969 MBmodel-00009.safetensors 969 MBmodel-00005.safetensors 969 MBmodel-00010.safetensors 959 MBmodel-00011.safetensors 609 MBmodel-00002.safetensors 8.3 kB