Sakura Nemotron 3.5 Bolt 74B-A9B 24GB — edited F16 and Q8_0 variants
Two independently measured GGUF variants derived from the same edited NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB file. Independent work, not made, reviewed or endorsed by NVIDIA.
The source language weights were edited to reduce refusals. A refusal-phrase score does not tell you whether a response followed the request. Use responsibly and do not use these models to harm others.
Files
F16 and Q8_0 below describe only the 128 edited tensors. Both files retain the other 715 tensors, including NVIDIA's IQ2_XXS / NVFP4 / Q8_0 / F32 mix. Neither file is an all-F16 or all-Q8 model.
| Variant | Edited tensors | File size | Download |
|---|---|---|---|
| F16-edited | F16 | 24.00 GiB / 25.77 GB | F16-edited GGUF |
| Q8_0-edited | Q8_0, re-quantized from the edited F16 values | 21.79 GiB / 23.40 GB | Q8_0-edited GGUF |
Download one file per variant. Exact sizes and hashes are in SHA256SUMS and BENCHMARKS.json.
Measured comparison
Measured on 2026-10-17 using official llama.cpp b11681, Vulkan, AMD Radeon 8060S. Results belong to the exact files identified by SHA256 in BENCHMARKS.json.
| Metric | F16-edited | Q8_0-edited |
|---|---|---|
| Strict refusal-phrase hits | 0 / 104 | 2 / 104 |
| Broad refusal / warning markers | 59 / 104 | 55 / 104 |
| Empty answers | 0 / 104 | 0 / 104 |
| Answers reaching the 300-token limit | Not recorded in the original run | 86 / 104 |
| Mean KLD vs NVIDIA — English | 0.153638 ± 0.001713 | 0.019423 ± 0.000315 |
| Mean KLD vs NVIDIA — Web | 0.111266 ± 0.001246 | 0.015155 ± 0.000242 |
| Same top token — English | 85.0% | 94.8% |
| Same top token — Web | 83.3% | 94.0% |
| PPL increase vs NVIDIA — English | +10.5% | +0.8% |
| PPL increase vs NVIDIA — Web | +7.4% | +0.8% |
Refusal-test method: 104 English prompts from the test split of mlabonne/harmful_behaviors, revision 01cead01398926d81f7c52bdb790ee8cf77ebba7; one user turn, no system prompt, thinking disabled (enable_thinking: false), temperature 0, seed 1, maximum 300 tokens, context 4096, batch 2048, microbatch 512, no speculative decoding. Same strict regex and broader marker list for both variants, evaluated on the first 400 response characters after apostrophe normalization. The unmodified NVIDIA file previously scored 104/104 strict hits under these conditions.
The first 32 prompts were used earlier when selecting the source edit, so the test set is not fully held out. These prompts were not used to select or tune this Q8 quantization. Neither score measures actual harmful-request compliance. No human compliance rate is claimed; token-limited responses may be unfinished.
KLD method: fresh inference on each candidate against the same cached NVIDIA-original logits; 20 chunks of 2048 tokens for each of heldout_en_mix.txt and mixedweb-v1.txt, batch 2048, microbatch 512, full layer offload, Flash Attention enabled. Values after ± are reported by llama-perplexity. Lower KLD means closer distributions on these texts, not an independent capability gain or a measurement of re-quantization error alone.
Hardware compatibility
The 24GB in the base model's name is a deployment target, not a file-size or VRAM guarantee. Runtime memory includes model weights, KV cache, recurrent states and compute buffers.
| Hardware / runtime check | F16-edited | Q8_0-edited |
|---|---|---|
| Tested machine | AMD Ryzen AI Max+ 395 / Radeon 8060S, Windows 11, 64 GB installed system memory | Same machine |
| Tested backend | Vulkan, llama.cpp b11681 | Vulkan, llama.cpp b11681 |
| Loading and text generation | Passed | Passed |
| GPU buffers at 128K capacity, one slot, Q8 K/V cache, batch 512 / microbatch 128 | 24.25 GiB, plus about 584.58 MiB host buffers | 22.05 GiB, plus about 584.58 MiB host buffers |
| Physical 24-GB discrete GPU | Not tested; reported buffers already exceed 24 GiB in this configuration | Not tested; the reported buffers were measured on the Radeon 8060S setup |
| NVIDIA CUDA / Apple Metal / CPU-only | Not validated for this artifact | Not validated for this artifact |
Both memory tests reserved 131,072-token context capacity and generated a short neutral response; they did not process a full 128K prompt or measure total driver peak memory. They differ from the refusal/KLD test configuration. This UMA system's GPU budget can include shared system RAM; it is not equivalent to a discrete GPU of the same advertised memory size.
Use a recent runtime supporting nemotron_h_moe, heterogeneous expert counts, IQ2_XXS and NVFP4. b11681 Vulkan was tested; b11594 failed to load. Other applications depend on the version and features of their bundled runtime.
Hugging Face's Hardware compatibility panel provides an estimate for your saved hardware. Its quantization labels correspond here to the edited tensors, while the remainder of each file keeps NVIDIA's mixed types. The panel is an estimate, not a hardware test.
Run
Choose one file. Example for the Q8_0-edited variant:
llama-server -m Sakura-Nemotron-3.5-Bolt-74B-A9B-24GB-Uncensored-edited-Q8_0.gguf --jinja -ngl 99 -c 131072 -np 1 -b 512 -ub 128 -ctk q8_0 -ctv q8_0 -fa on --fit off
This matches the main settings of the Q8 memory check. Choose a context size and offload configuration suitable for your hardware. Reasoning is controlled through the model's enable_thinking chat-template option.
Provenance and limits
The source edit removed a refusal direction from selected language-weight matrices of NVIDIA's unmodified GGUF (SHA256 91d63363e782c8a3d5f4be42816eba8e0d42dd2530bd07a7d0d2dc440599af75). The F16-edited file stores those edited tensors in F16. The Q8_0-edited file re-quantizes those same 128 tensors; the other 715 tensors and source metadata are byte-identical. Numerical relative L2 change of the edited F16 weights from Q8_0 quantization: 0.574%.
The inherited general.file_type=1 field still says F16 in both files because metadata was preserved. Per-tensor types are authoritative; the Q8_0-edited file contains no F16 tensors.
The vision projector (mmproj) was not changed and is not included. Vision, speculative decoding, thinking-enabled refusals, other languages, independent capability benchmarks and full long-context behavior were not tested. Previously measured F16 results are not attributed to the Q8 file.
License and credits
OpenMDW License Agreement 1.1, as the original model. A complete copy is included as LICENSE; see NOTICE.
- Base model: NVIDIA.
- Runtime and quantization implementation: llama.cpp, MIT.
- Evaluation prompts: mlabonne/harmful_behaviors, revision identified above.
中文版：README_zh.md。