Bielik-11B-v3.1-NVFP4-mix
Mixed-precision NVFP4 quantization of speakleash/Bielik-11B-v3.1-Instruct, made with GPTQ.
This is an unofficial community quantization. The model, its training and all credit belong to SpeakLeash and ACC Cyfronet AGH. It is released under the base model's Apache 2.0 license. For model details, the chat template and intended use, see the original model card.
- Format: compressed-tensors, served by vLLM. Native W4A4 kernels need a Blackwell GPU. It was tested on an RTX 5070 Ti (sm_120) and a DGX Spark / GB10 (sm_121).
- Size: 6.82 GiB, against 20.8 GiB for BF16 and 6.20 GiB for the official NVFP4.
- Recipe:
- Most linear layers use NVFP4: W4A4, group size 16, FP8 E4M3 scales.
- The attention projections (q/k/v/o) of the 36 most sensitive layers use FP8 instead: W8A8, dynamic per-token activations. These are layers 0–20, 25–38 and 40.
lm_headstays in BF16.
- Intended use: an NVFP4 model for serving several concurrent requests with DFlash speculative decoding, with quality noticeably closer to BF16 than the official NVFP4 checkpoint.
Usage (vLLM)
Bielik v3.1 uses its own tool-call and reasoning format (<tool_call>…</tool_call>, <think>…</think>). To get structured tool_calls and a separate reasoning field from the OpenAI-compatible API, load the Bielik parsers from speakleash/bielik-tools, as recommended in the base model card (vLLM 0.24 or newer):
git clone https://github.com/speakleash/bielik-tools.git
vllm serve kurcontko/Bielik-11B-v3.1-NVFP4-mix \
--max-model-len 8192 \
--speculative-config '{"method":"dflash","model":"speakleash/Bielik-11B-v3.1-DFlash2","num_speculative_tokens":15}' \
--attention-backend flash_attn \
--enable-auto-tool-choice \
--tool-parser-plugin ./bielik-tools/tools/bielik_vllm_tool_parser.py \
--tool-call-parser bielik \
--reasoning-parser-plugin ./bielik-tools/tools/bielik_vllm_reasoning_parser.py \
--reasoning-parser bielik \
--exclude-tools-when-tool-choice-none
- The parsers only post-process the API response; they do not change what the model generates. All quality and speed numbers below were measured without them, through
/v1/completionson token ids. - The parser flags follow the base model card. We have not yet tested them together with DFlash speculative decoding on this checkpoint.
--max-model-len 8192matches our measurements. The base model card uses 32768; raise it if you need longer context and have the memory.- Load exactly one tool parser and one reasoning parser (both register the name
bielik).
The chat template is identical to the base model's. By default it injects a built-in system prompt of about 1,000 tokens, including the current date. Disable it with --default-chat-template-kwargs '{"provider_system_prompt": false}' or per request. reasoning_effort defaults to none; set it per request (low, medium, high, max) to enable thinking.
Quality vs BF16
All quality numbers below compare against speakleash/Bielik-11B-v3.1-Instruct in BF16, on held-out data. 95% CIs come from 2000 bootstrap draws, resampling over prompts, documents or questions.
Fidelity
| this model | ours, uniform NVFP4-GPTQ | official NVFP4 | |
|---|---|---|---|
| KL(BF16‖Q), nats/token | 0.0310 [0.0290, 0.0333] | 0.0430 [0.0406, 0.0458] | 0.0593 [0.0562, 0.0626] |
| top-1 agreement with BF16 | 0.9615 | 0.9557 | 0.9473 |
| Polish web ΔNLL, nats/token | +0.0274 | +0.0287 | +0.0322 |
- KL is a Monte-Carlo estimate. It uses 565 BF16 responses (T=1, up to 384 tokens) to 283 held-out prompts across 9 domains: Polish instructions and Aya data in pl/en/de/fr/es/uk, plus code and math. The model was scored with
prompt_logprobsin vLLM. - Web ΔNLL is measured on 200 held-out FineWeb-2
pol_Latndocuments, up to 2048 tokens each. - Paired difference vs the official NVFP4 (this model minus official): KL −0.0283 [−0.0306, −0.0260] and web NLL −0.0048 [−0.0078, −0.0013]. The model is better in all 9 prompt domains.
Tasks
These are zero-shot results from lm-eval 0.4.13 (loglikelihood). Global-MMLU-PL is subsampled to 50 questions per subject.
| task | BF16 | this model | official NVFP4 |
|---|---|---|---|
| belebele_pol_Latn (acc) | 85.8 | 84.7 | 83.2 |
| arc_challenge_mt_pl (acc) | 49.1 | 47.7 | 47.4 |
| include_base_44_polish (acc) | 66.6 | 66.8 | 64.6 |
| global_mmlu_full_pl (acc) | 63.6 | 62.1 | 60.7 |
| polemo2_in (f1) | 74.8 | 75.8 | 75.2 |
| polemo2_out (f1) | 67.4 | 70.2 | 66.4 |
The paired difference vs the official NVFP4, pooled over the 4 knowledge tasks (5470 questions), is +1.26 pp [+0.48, +2.14]. PolEmo2 is noisy and should be read with care: differences of a few points in either direction appear across all quantizations.
Speed
Speed was measured on 80 held-out prompts with greedy decoding, up to 512 new tokens, a BF16 KV cache and max-model-len 8192. "c1" is per-request decode speed with a single client. "c4" is total throughput with 4 concurrent clients. Mean acceptance length with DFlash2 is 3.71 tokens per step.
| hardware | config | c1, tok/s | c4, tok/s |
|---|---|---|---|
| RTX 5070 Ti 16 GB (vLLM 0.33.0) | no speculation | 75.9 | 269 |
| + DFlash2 (15 tokens) | 188 | 582 | |
| + DFlash v1 (15 tokens) | 186 | 587 | |
| DGX Spark / GB10 (vLLM 0.32) | no speculation | 27.9 | 104 |
| + DFlash2 (15 tokens) | 72.8 | 252 |
The FP8 attention layers cost about 6% of speed compared with uniform NVFP4: 201 vs 188 tok/s at c1 on the RTX 5070 Ti.
On GB10, the weight-only NVFP4A16 is as fast as W4A4, because decode there is memory-bound. If you serve on GB10, an NVFP4A16 checkpoint gives better quality at the same speed.
How it was made
Tool and settings. llm-compressor 0.15, GPTQ with the sequential pipeline, dampening 0.01 and static activation ordering.
Calibration data: 809 sequences.
- 609 chat samples. The prompts come from Polish Alpaca/Dolly, the CohereLabs Aya dataset (pl, en, de, fr, uk, es) and synthetic code and math tasks. The answers were generated by the BF16 model at T=0.3, up to 768 tokens.
- 200 FineWeb-2
pol_Latndocuments, up to 2048 tokens each. - The calibration and evaluation sets are disjoint.
Choosing the FP8 layers.
- A leave-one-in sensitivity scan measured the damage of each layer group on its own. Each attention or MLP group of a uniform NVFP4-GPTQ model was swapped, one at a time, into the BF16 model, and the exact full-vocabulary KL was measured on 64 held-out samples.
- Groups were then picked greedily by KL removed per extra byte, under a +0.62 GiB budget.
- Per parameter, attention groups were about 4× more sensitive than MLP groups. The selected groups account for 35% of the summed single-group KL.
The llm-compressor recipe is included as recipe.yaml, and the layer selection as quant_recipe.json.
Limitations
- Calibration matches the KL test set in style. Most calibration samples are BF16-generated chat answers, and KL is measured on the same kind of text, although the prompts differ. This favours chat fidelity. On raw web text the gain over the official checkpoint is smaller (see web ΔNLL above).
- Greedy outputs are not identical to BF16. Generations diverge from the BF16 output after about 30 tokens on average. This is expected for 4-bit activations.
- Hardware support is limited. This checkpoint was only tested with vLLM on the Blackwell GPUs listed above.