NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF

NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 is NVIDIA's full-precision small language model, post-trained from Google's Gemma-4-E4B-it (8B total, 4B effective parameters), aimed at fast local voice and device assistants, interactive personas and embedded or edge interfaces. It keeps Gemma's conversational strengths and improves multi-turn tool use, instruction following and input grounding, with low memory use and low time to first token. Post-training was multi-environment reinforcement learning (GRPO, asynchronous, with NeMo RL and NeMo Gym) over instruction following, multi-step tool use, multi-turn conversation and structured-output environments. The model is text-in, text-out, with up to 64K tokens of input and output. It has chat-template switches for thinking (enable_thinking) and for a conversational_mode that gives concise, less formatted replies suited to spoken use, plus a documented client for capping the thinking budget at runtime. Against Gemma-4-E4B-it, Qwen3.5-4B and Nemotron-3-Nano-4B, measured by NVIDIA under NeMo Gym, it leads the BF16 group on most rows, including MultiChallenge (53.10), RoleBench (67.05), IHEval (84.79), IFBench (63.79) and RelyToolBench (63.16). Qwen3.5-4B is ahead on Tau2-Bench (75.06 vs 55.10), RULER at 64K, LongBench v2, BFCL-v3 and ToolSandbox, and Gemma leads EQ-Bench v3. The card also reports Q4_K_M results for the companion 4 GB build, which stay close to the BF16 scores. It is served with vLLM (v0.31.0) using the Gemma-4 tool and reasoning parsers, and a companion DFlash drafter is offered for speculative decoding. The card says this BF16 release is intended mainly for customization, post-training, quantization and research rather than direct production inference, and points to the Q4_K_M build for deployment. It is released under the OpenMDW-1.1 license, and the card notes the model was post-trained on Gemma 4.

Model Files

Q4_K_M Test Run

Tested NVIDIA-Nemotron-3.5-Bolt-E4B.Q4_K_M.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading.

Performance Metrics

Test Shots

Run Inference

./build/bin/llama-cli \
  -m /path/to/models/NVIDIA-Nemotron-3.5-Bolt-E4B.Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

---

Snapshots (5)

---

Quantized using llama.cpp by prithivMLmods