Nemotron-3.5-Bolt-E4B GSQ-RCO GGUF
Mixed-precision GSQ-RCO GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, optimized for efficient, high-fidelity local inference in llama.cpp.
The model supports a context length of up to 65,536 tokens (64K).
Benchmarks
Why GSQ-RCO?
Standard quantization severely degrades hybrid sliding-attention models with shared Key-Value caches, multimodal projections, and speculative draft targets.
GSQ-RCO solves this through two targeted techniques:
- RCO: Preserves all 54 shared
k_proj,v_proj, andnormtensors across layers 24–41 in high precision (Q5_K/Q6_K), maintaining structural attention geometry, visual projection alignment, and DFlash draft compatibility. - GSQ: Optimizes per-layer precision guided by Hessian sensitivity calibration.
GGUF Files
| File | bpw | Size |
|---|---|---|
| Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ3_M.gguf (Recommended) | 3.75 | 3.98 GB |
| Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ4_XS.gguf | 4.25 | 4.80 GB |
| Nemotron-3.5-Bolt-E4B-GSQ-RCO-Q5_K_M.gguf | 5.25 | 5.87 GB |
| Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf (Official Draft) | - | 737 MB |
| Nemotron-3.5-Bolt-E4B-DFlash-Q8_0.gguf (Compact Draft) | - | 399 MB |
| mmproj-BF16.gguf (Official Multimodal Projector) | - | 991 MB |
| mmproj-Q8_0.gguf (Compact Multimodal Projector) | - | 559 MB |
Usage & Execution Examples
Nemotron-3.5-Bolt-E4B GSQ-RCO supports text inference, accelerated speculative decoding via DFlash, and vision/audio multimodal understanding via mmproj.
1. Speculative Decoding (+DFlash)
Up to 7 tokens parallel drafting for 100+ tok/s generation at 64k context:
llama-server \
-m Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ3_M.gguf \
-md Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 7 \
-c 65536 \
-ngl 99 -ngld 99
2. Multimodal Vision & Audio (+mmproj)
Enable visual understanding and audio processing with the companion multimodal projector:
llama-server \
-m Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ3_M.gguf \
--mmproj mmproj-BF16.gguf \
-c 65536 \
-ngl 99
Interactive CLI image query:
llama-cli \
-m Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ3_M.gguf \
--mmproj mmproj-BF16.gguf \
--image sample.png \
-p "Describe what is written in this image." \
-ngl 99
3. Combined Mode (+DFlash +mmproj)
Maximum performance: High-accuracy multimodal processing accelerated with DFlash speculative drafting:
llama-server \
-m Nemotron-3.5-Bolt-E4B-GSQ-RCO-IQ3_M.gguf \
--mmproj mmproj-BF16.gguf \
-md Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 7 \
-c 65536 \
-ngl 99 -ngld 99
Attribution & License
- Distributed under the OpenMDW-1.1 License, matching upstream nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.
- Upstream model repository: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.