How to Run NVIDIA Nemotron 3.5 Super VL 120B-A12B Without Refusals Locally

NVIDIA's refusal behaviour projected out of the released weights, baked into Atomic Dynamic GGUF builds and measured against the original.

  • On held-out JailbreakBench prompts refusals fall from 95.1% to 0%, on Russian red-teaming prompts from 76% to 2%, and with thinking on from 30 of 30 to 2 of 30. MMLU stays where it was: 85.05% before, 85.10% after (2000 questions, McNemar p = 1.0). Tool calls and long context are unaffected; no empty or looping replies.
  • The refusal is written into the residual stream by every kind of block: the edit covers the Mamba-2 outputs, the attention outputs, the shared experts and the latent projection through which all 512 routed experts write.
  • Same Atomic Dynamic layouts and importance matrix as our Nemotron 3.5 Super quants; each file is the size of the original build with the same name.

The builds

Build On disk Mean KLD Same top-1
AD-Q4_K_S 86.9 GB 0.0125 95.30%
AD-IQ3_XXS 65.3 GB 0.0397 91.90%
AD-Q2_0_L 55.8 GB 0.0763 88.94%

KLD and top-1 are against the original, unmodified BF16 model on our held-out neutral text (24 chunks of 4096 tokens), so they hold the edit and the quantization together. For scale, our plain builds of the original model with the same layouts are 0.0082, 0.0352 and 0.0722 (on 95 chunks): the edit itself adds about 0.004.

The baked files themselves, on the held-out prompts: AD-Q4_K_S refused 0 of 81 English, 3 of 100 Russian and 1 of 250 XSTest prompts; AD-IQ3_XXS 0, 2 and 1; AD-Q2_0_L 0, 5 and 0.

The vision projector (mmproj-*.gguf) is the original model's, unchanged; image input was not tested on these files.

Running it

hf download AtomicChat/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Uncensored-GGUF \
  --include "NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Uncensored-AD-IQ3_XXS/*" --local-dir .
llama-server -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Uncensored-AD-IQ3_XXS/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Uncensored-AD-IQ3_XXS-00001-of-00043.gguf \
  -ngl 99 -c 32768 --jinja

The chat template is inside every file. Upstream llama.cpp with nemotron_h_moe support runs them; no patch is needed.

How it was made

  1. Direction. Difference of means of the residual stream at the last prompt token, thinking off, chat template applied: 416 harmful against 416 harmless English prompts and the same in Russian, on the released weights. The component along the harmless mean is removed. One direction, measured at the input of block 66 of 88.
  2. Edit. W' = W - r rᵀW on the 128 matrices that write into the residual stream: the output projection of all 40 Mamba-2 blocks and of the 8 attention blocks, the down projection of the 40 shared experts, and the 40 latent up projections through which every routed expert writes. The router, the token embeddings and the multi-token-prediction head are untouched.
  3. Choice. The recipe was set from a rehearsal on a smaller model of the same family and confirmed here against an alternative (block 50) and a random direction of the same size as the control, on validation prompts and then on every check below.
  4. Bake. The same edit applied to the BF16 weights in f32 and rounded once, then quantized with the importance matrix and the layouts of our original quants.

What changed, and what did not

Original against this edit on the released weights (BF16, the edit applied as an adapter). The refusal sets are held-out prompts never used to pick anything.

Original This edit Random direction, same size
JailbreakBench harmful behaviours (81), refused 95.1% 0.0% 93.8%
Russian red-teaming prompts (100, aya), refused 76.0% 2.0% 75.0%
XSTest safe prompts (250), refused 5.2% 0.4% 4.0%
With thinking (30 prompts, 3072 tokens), refused 30 of 30 2 of 30 -
Thinking left unclosed at 3072 tokens 0 of 30 0 of 30 -
Empty or looping replies 0 0 0
MMLU, 2000 questions 85.05% 85.10% (McNemar p = 1.0) 85.10%
Tool calls (20) 20/20 valid 20/20 valid -
Needle at 30k tokens (3 depths) 3/3 3/3 -

Validation prompts (104 harmful, 104 harmless per language), replies of up to 64 tokens, the check of the recipe on the released weights: 1.9% of English and 3.0% of Russian harmful prompts refused.

Refusal is counted by the opening of the reply, so it is indicative, not a judge.

Limitations

  • With thinking on, 2 of 30 prompts were still refused; our gate allows fewer. We publish because every other check passed and the difference from zero is inside the noise of 30 prompts, but the thinking mode is not fully cleared.
  • The refusal count reads the opening of each reply; a soft refusal phrased as an answer would be missed.
  • MMLU and the thinking check use fixed question sets; they show a large change, not a small one.
  • We publish the baked files only. The same edit as a LoRA adapter needs a llama.cpp patch on this architecture (upstream applies no LoRA to the latent projections), so it is not offered here.

Responsible use

This model has its learned refusal behaviour removed, so it will answer requests the original declines. Nothing about it makes the output safe, correct or lawful, and any deployment needs its own access controls and policy enforcement.

License and credits

OpenMDW-1.1, as the original. Base model Nemotron 3.5 Super VL 120B-A12B by NVIDIA. Direction, edit, quantization and evaluation by AtomicChat. The method follows Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).