NVIDIA-Nemotron-3.5-Bolt-E4B GGUF (Tere Labs)

imatrix GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, NVIDIA's post-trained version of Gemma-4-E4B-it, made by Tere Labs. This is an independent community quantization, not an official NVIDIA release. NVIDIA's own GGUF is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB.

Two files and one imatrix. Every KLD, top-1 and tool-call number below was measured the same way, against a BF16 GGUF converted from the original weights.

Files

bpw is file size over all 7.52B weights in the file, including the per-layer embeddings. NVIDIA's 8B (4B effective) counts the full BF16 checkpoint, including the vision and audio encoders that these text-only files leave out.

Running

Gemma 4 support is in llama.cpp main, no fork or PR needed. These files were converted, quantized and tested with tag b11790 (0efe8e4).

llama-server -hf terelabs/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF:Q6_K --jinja

• Both files load the embedded chat template with --jinja. We checked that each one returns a correct function call through the OpenAI-compatible API.
• With --jinja, llama-server turns thinking on by default for this model (--reasoning auto), although the template itself defaults enable_thinking to false. To turn it off, start the server with --reasoning off or send "chat_template_kwargs": {"enable_thinking": false} per request.
• NVIDIA recommends temperature 1.0 and top_p 0.95 in both thinking modes.
• context_length in these files is 65536, as in the BF16 config.json and NVIDIA's model card (up to 64K tokens).
• Text in, text out, as in NVIDIA's model card. The BF16 checkpoint also contains Gemma 4's vision and audio encoders. They were not converted, so these files are text-only.

Recipe

llama-quantize --imatrix imatrix.gguf with base type Q6_K or Q5_K_M and these overrides:

Why these tensors:

• The model has 42 layers. The last 18 reuse K/V from earlier layers (shared KV), so their attn_k/attn_v weights are not used, and we store them at Q4_K. The K/V weights that are used get Q8_0.
• Per-layer embeddings (per_layer_token_embd, 2.82B of the 7.52B weights) enter every layer through proj and inp_gate. The default llama-quantize layout gives these two the file's base type, so NVIDIA's Q4_K_M stores them at Q4_K. Raising them to Q8_0, together with the K/V weights, cut KLD by about a quarter at both Q4_K_M and Q5_K_M (ablations below).
• unsloth's UD files use the same ideas: proj at Q8_0 and inp_gate at Q6_K or Q8_0 in all three, and their UD-Q4_K_XL stores the unused K/V of layers 24-41 at Q2_K and per_layer_token_embd at IQ4_NL. Our Q6_K differs from UD-Q6_K_XL mainly in per_layer_token_embd (Q6_K instead of Q8_0) and the unused K/V (Q4_K instead of Q8_0), which is where its 0.7 GB saving comes from.

imatrix

About 600k tokens, ctx 512, 1187 chunks, no partial-data warnings:

• ~300k tokens of tool-calling dialogues from NousResearch/hermes-function-calling-v1 (Apache-2.0), rendered through the model's own chat template
• ~150k tokens of code from permissive repositories at pinned commits: requests, flask, express, ripgrep, gin, nlohmann/json
• ~150k tokens of wikitext train

No overlap with BFCL. The wikitext part is from the train split, and KLD and top-1 below are measured on the wikitext-2 test split, so our calibration text is closer to that evaluation text than unsloth's or NVIDIA's may be.

Ablations

KLD against BF16, same evaluation as below. These are separate runs and are not in comparison.csv.

• With NVIDIA's layout our imatrix lands where NVIDIA's file does (0.177 against 0.172), so against NVIDIA's file the gain comes from the tensor layout, not from the calibration data. unsloth's layouts are close to ours, so the differences to their files come from calibration and noise, not layout.
• proj, inp_gate and the K/V weights at Q8_0 cut KLD by 25% at Q5_K_M (0.064 to 0.048) for about 65 MB. That run set attn_k/v to Q8_0 in all 42 layers. The released files keep the 18 unused ones at Q4_K, so the same change costs them about 38 MB.
• Embeddings at Q6_K instead of Q8_0 cost nothing measurable (0.177 to 0.175).
• The released Q5_K_M stores per_layer_token_embd at IQ4_NL. That makes it 0.73 GB smaller than the 5.83 GB variant, for KLD 0.049 against 0.048. token_embd at Q8_0 does not help it (0.051 against 0.049, within noise).
• IQ4_XS (4.61 GB) is further from BF16 than NVIDIA's Q4_K_M (0.200 against 0.172), so it is not included.

Comparison with other GGUFs of this model

All rows measured by us against the same BF16 reference. Lower KLD is better, higher agreement is better. "n/m" means not measured.

Bold marks the best point estimate in each column. At the Q6 level the KLD and top-1 gaps, and all tool-call and BFCL gaps, are within measurement error.

Measurement error: KLD mean ±0.0005 to ±0.003 and top-1 ±0.14 to ±0.23 pp (one standard error); same tool call ±1.8 to ±2.5 pp and BFCL ±4.1 to ±4.2 pp (95% CI, 300 tasks).

What the table supports:

• Q6_K is about 10x closer to BF16 than NVIDIA's official GGUF at nearly the same size: KLD 0.0174 against 0.1718, top-1 95.0% against 84.5%, for 6.40 GB against 6.18 GB.
• Q6_K matches unsloth UD-Q6_K_XL and is 0.7 GB smaller: KLD 0.0174 against 0.0185, 6.40 GB against 7.11 GB.
• Q5_K_M is 1.1 GB smaller than NVIDIA's official GGUF and 3.5x closer to BF16: 5.07 GB against 6.18 GB, KLD 0.049 against 0.172.
• Q5_K_M is 0.8 GB smaller than unsloth UD-Q5_K_XL and slightly further from BF16: KLD 0.049 against 0.046, top-1 91.6% against 91.9%.
• At around 5 GB, unsloth UD-Q4_K_XL is the better file: slightly closer to BF16 than our Q5_K_M (KLD 0.046 against 0.049) and 0.3 GB smaller.
• Tool calls: both our files repeat BF16's calls in 97% of tasks, NVIDIA's file in 94.7%, unsloth UD-Q4_K_XL in 97.3%. The differences are within the 95% intervals above.
• No measured file differs significantly from BF16 on BFCL: the four quantized files we ran land at 83.3-84.0% against 84.0% for BF16 (McNemar, all p ≥ 0.69). With 300 tasks, differences smaller than about 4 pp cannot be detected.

Method

KLD and top-1:

• llama-perplexity --kl-divergence against BF16 logits
• wikitext-2 test, ctx 2048, 24 chunks

Tool calling:

• 300 BFCL v3 tasks (simple, multiple, parallel, parallel_multiple, live_simple, live_multiple, irrelevance, live_irrelevance), the same set and seed as our LFM2.5 and Flint releases
• prompts rendered with NVIDIA's chat template from the BF16 release, thinking off (the template default), greedy decoding. A repeat BF16 run matched on 30 of 30 tasks.
• "same tool call": function name and all arguments match BF16
• "BFCL correct": a simplified BFCL AST checker against the reference answers

Per-file raw numbers, including per-category BFCL accuracy, are in comparison.csv in this repository.

License and notices

These files are quantized derivatives of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16. NVIDIA's notice for that model, retained as required by OpenMDW-1.1:

Governing Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1. This model has been post-trained on the Google Gemma 4 model, and use of the Google Gemma 4 model is governed by Apache License, Version 2.0

Model developer: NVIDIA Corporation. Base model: google/gemma-4-E4B-it by Google DeepMind.

• LICENSE: the OpenMDW-1.1 agreement.
• LICENSE-APACHE-2.0: the Apache License 2.0 that governs the Gemma 4 base model.

Changes made by Tere Labs:

1. converted the language model from safetensors to GGUF with llama.cpp b11790. The vision and audio encoders of the BF16 checkpoint were not converted, so these files are text-only;
2. weights quantized with llama.cpp using the imatrix and recipe above;
3. added imatrix.gguf, computed by Tere Labs from the calibration data listed above.

The embedded chat template is byte-identical to chat_template.jinja of the BF16 release.

For NVIDIA's evaluation results, intended use and the Bias, Explainability, Safety & Security and Privacy subcards, see the original model card. NVIDIA and Nemotron are trademarks of NVIDIA Corporation. Gemma is a trademark of Google LLC. This repository is not affiliated with or endorsed by NVIDIA or Google.