Qwen3.5-4B-FP8
Block-wise FP8 (e4m3) quantization of Qwen/Qwen3.5-4B in the exact format of Qwen's own official *-FP8 releases (quant_method: fp8, weight_block_size [128,128], dynamic activations, fp32 weight_scale_inv scales).
Qwen published official FP8 weights for Qwen3.5-27B/35B-A3B/122B/397B — but not for the 4B, and it kept being asked for (#13, #24, #27). So here it is.
- Size: 5.65 GB (from ~8.8 GB BF16)
- No calibration — this is a deterministic weight transform. The conversion script is included in this repo (
quantize_fp8_block.py), as requested in #24.
Run it
Serve (tested — vLLM 0.19.0, RTX 5070 12 GB; vLLM downloads the weights automatically):
vllm serve Hob-forge/Qwen3.5-4B-FP8 --max-model-len 4096 --gpu-memory-utilization 0.75
Or in Python:
from vllm import LLM, SamplingParams
llm = LLM(model="Hob-forge/Qwen3.5-4B-FP8", max_model_len=4096, gpu_memory_utilization=0.75)
out = llm.chat(
[{"role": "user", "content": "Hello!"}],
SamplingParams(temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5, max_tokens=512),
)
print(out[0].outputs[0].text)
To just fetch the weights:
hf download Hob-forge/Qwen3.5-4B-FP8
Hardware: FP8 GEMM needs a recent NVIDIA GPU (Ada / Hopper / Blackwell, sm89+); ~6 GB VRAM for weights plus KV cache — a 12 GB card runs it comfortably at 4K context. On an 8 GB card, CPU, or modest RAM? FP8 is not the right artifact for you — use our GGUF edition instead: Hob-forge/Qwen3.5-4B-Instruct-GGUF.
Module treatment
Mirrors Qwen/Qwen3.5-27B-FP8 exactly:
| modules | |
|---|---|
| FP8 block-128 | language self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, linear_attn.{in_proj_qkv, in_proj_z, out_proj}, MTP-block attention + MLP projections |
| kept BF16 | embeddings (tied lm_head), linear_attn.{conv1d, in_proj_a, in_proj_b}, the entire vision tower, mtp.fc, all norms |
207 weight tensors quantized, 531 kept bit-identical to upstream.
Verification
- Every quantized tensor dequantized and compared against upstream BF16: max block-dequant relative error 3.6%, mean 2.8% — the expected e4m3 envelope.
- All passthrough tensors verified bit-identical to upstream.
- Smoke-tested end-to-end with vLLM 0.19.0 on an RTX 5070 12 GB (Blackwell, WSL2): greedy completions and chat-template generation are coherent; thinking mode works.
A/B against BF16 (same box, same inputs, vLLM 0.19.0, teacher-forced):
| measurement | BF16 | FP8 | delta |
|---|---|---|---|
| PPL, prose (Austen excerpt, 4776 tok) | 15.141 | 15.174 | +0.22% |
| PPL, prose (Shelley excerpt, 3741 tok) | 10.173 | 10.256 | +0.81% |
| PPL, code (Python, 2800 tok) | 2.859 | 2.864 | +0.19% |
| 12-item exact-answer QA (greedy) | 12/12 | 12/12 | parity |
Greedy 64-token continuations match BF16 token-for-token 61% of the time (7/20 prompts identical throughout) — normal for any quantization: near-tie top-2 tokens flip and paths diverge. Every diverged continuation was manually inspected and is coherent. This is a small probe, not a benchmark suite — for production decisions run your own evals.
Sampling parameters
Use upstream's recommendations (from the Qwen3.5-4B card):
- Thinking, general:
temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5 - Thinking, precise coding:
temperature=0.6, top_p=0.95, top_k=20 - Instruct, general:
temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5
Tested only on Blackwell (RTX 5070) under vLLM; other stacks (SGLang, transformers fine-grained-FP8) load this format but are untested by us — reports welcome.
NVFP4
Also requested in #27 — under evaluation. NVFP4 needs a calibration pass and currently serves only on Blackwell-class hardware; if we can produce and validate one honestly, it will appear as a sibling repo.
Who we are
Hob Forge — small household spirits for small machines. We set up small models correctly for small GPUs and modest RAM.
Quantized with the included script; Apache-2.0, upstream LICENSE retained.