Mixed-precision NVFP4 quantization of speakleash/Bielik-11B-v3.1-Instruct, made with GPTQ.
This is an unofficial community quantization. The model, its training and all credit belong to SpeakLeash and ACC Cyfronet AGH. It is released under the base model's Apache 2.0 license. For model details, the chat template and intended use, see the original model card.
lm_head stays in BF16.Bielik v3.1 uses its own tool-call and reasoning format (<tool_call>…</tool_call>, <think>…</think>). To get structured tool_calls and a separate reasoning field from the OpenAI-compatible API, load the Bielik parsers from speakleash/bielik-tools, as recommended in the base model card (vLLM 0.24 or newer):
git clone https://github.com/speakleash/bielik-tools.git
vllm serve kurcontko/Bielik-11B-v3.1-NVFP4-mix \
--max-model-len 8192 \
--speculative-config '{"method":"dflash","model":"speakleash/Bielik-11B-v3.1-DFlash2","num_speculative_tokens":15}' \
--attention-backend flash_attn \
--enable-auto-tool-choice \
--tool-parser-plugin ./bielik-tools/tools/bielik_vllm_tool_parser.py \
--tool-call-parser bielik \
--reasoning-parser-plugin ./bielik-tools/tools/bielik_vllm_reasoning_parser.py \
--reasoning-parser bielik \
--exclude-tools-when-tool-choice-none
/v1/completions on token ids.--max-model-len 8192 matches our measurements. The base model card uses 32768; raise it if you need longer context and have the memory.bielik).The chat template is identical to the base model's. By default it injects a built-in system prompt of about 1,000 tokens, including the current date. Disable it with --default-chat-template-kwargs '{"provider_system_prompt": false}' or per request. reasoning_effort defaults to none; set it per request (low, medium, high, max) to enable thinking.
All quality numbers below compare against speakleash/Bielik-11B-v3.1-Instruct in BF16, on held-out data. 95% CIs come from 2000 bootstrap draws, resampling over prompts, documents or questions.
| this model | ours, uniform NVFP4-GPTQ | official NVFP4 | |
|---|---|---|---|
| KL(BF16‖Q), nats/token | 0.0310 [0.0290, 0.0333] | 0.0430 [0.0406, 0.0458] | 0.0593 [0.0562, 0.0626] |
| top-1 agreement with BF16 | 0.9615 | 0.9557 | 0.9473 |
| Polish web ΔNLL, nats/token | +0.0274 | +0.0287 | +0.0322 |
prompt_logprobs in vLLM.pol_Latn documents, up to 2048 tokens each.These are zero-shot results from lm-eval 0.4.13 (loglikelihood). Global-MMLU-PL is subsampled to 50 questions per subject.
| task | BF16 | this model | official NVFP4 |
|---|---|---|---|
| belebele_pol_Latn (acc) | 85.8 | 84.7 | 83.2 |
| arc_challenge_mt_pl (acc) | 49.1 | 47.7 | 47.4 |
| include_base_44_polish (acc) | 66.6 | 66.8 | 64.6 |
| global_mmlu_full_pl (acc) | 63.6 | 62.1 | 60.7 |
| polemo2_in (f1) | 74.8 | 75.8 | 75.2 |
| polemo2_out (f1) | 67.4 | 70.2 | 66.4 |
The paired difference vs the official NVFP4, pooled over the 4 knowledge tasks (5470 questions), is +1.26 pp [+0.48, +2.14]. PolEmo2 is noisy and should be read with care: differences of a few points in either direction appear across all quantizations.
Speed was measured on 80 held-out prompts with greedy decoding, up to 512 new tokens, a BF16 KV cache and max-model-len 8192. "c1" is per-request decode speed with a single client. "c4" is total throughput with 4 concurrent clients. Mean acceptance length with DFlash2 is 3.71 tokens per step.
| hardware | config | c1, tok/s | c4, tok/s |
|---|---|---|---|
| RTX 5070 Ti 16 GB (vLLM 0.33.0) | no speculation | 75.9 | 269 |
| + DFlash2 (15 tokens) | 188 | 582 | |
| + DFlash v1 (15 tokens) | 186 | 587 | |
| DGX Spark / GB10 (vLLM 0.32) | no speculation | 27.9 | 104 |
| + DFlash2 (15 tokens) | 72.8 | 252 |
The FP8 attention layers cost about 6% of speed compared with uniform NVFP4: 201 vs 188 tok/s at c1 on the RTX 5070 Ti.
On GB10, the weight-only NVFP4A16 is as fast as W4A4, because decode there is memory-bound. If you serve on GB10, an NVFP4A16 checkpoint gives better quality at the same speed.
Tool and settings. llm-compressor 0.15, GPTQ with the sequential pipeline, dampening 0.01 and static activation ordering.
Calibration data: 809 sequences.
pol_Latn documents, up to 2048 tokens each.Choosing the FP8 layers.
The llm-compressor recipe is included as recipe.yaml, and the layer selection as quant_recipe.json.
speakleash/Bielik-11B-v3.1-Instruct