NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara — GGUF (static quants)
🧬 Source model: spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara · base: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
📦 imatrix (i1) quants: spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-i1-GGUF
🛠️ Runs on llama.cpp (master)
📊 Scored on CaliperBench
💬 Discord
[!Note] Credit for the base model goes to NVIDIA — NVIDIA-Nemotron-3.5-Super-VL-120B-A12B, a 120B total / 12B active hybrid Mamba-Transformer latent-MoE with a RADIO vision tower and an MTP head. Credit for the method goes to p-e-w and the Heretic contributors (Arbitrary-Rank Ablation, Heretic 2.0). We ran the search and built the checkpoint (spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara); these are its quantizations.
[!Tip] Benchmarked on CaliperBench — a creative-writing benchmark scoring prose craft, roleplay and willingness rather than general intelligence. See this model's scores: caliperbench.com
[!Important] MTP included. NVIDIA's multi-token-prediction head ships as
mtp-*.ggufand speeds up decoding when loaded as the draft model (--spec-type draft-mtp; about 1.16x on our Q8_0 test). Built and tested with llama.cpp master; older builds may not load the vision projector or the MTP head.
Vision: this is a VL model. llama.cpp keeps the vision tower OUT of the main GGUF, so multimodal use needs
mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.ggufalongside your chosen quant (llama-server -m <quant>.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf). Text-only use needs just the quant.
This is a decensored version of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, made using Heretic v2.0.0.dev0
Arbitrary-Rank Ablation (ARA) parameters
| Parameter | Value |
|---|---|
| start_layer_index | 41 |
| end_layer_index | 54 |
| preserve_good_behavior_weight | 0.7358 |
| steer_bad_behavior_weight | 0.1777 |
| overcorrect_relative_weight | 0.9892 |
| neighbor_count | 2 |
Performance
| Metric | This model | Original model (nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) |
|---|---|---|
| Refusals | 6/100 | 100/100 |
| KL divergence | 0.1092 | 0 (by definition) |
Heretic's own evaluation: refusals are Heretic's keyword rule over 100 held-out prompts from mlabonne/harmful_behaviors; KL divergence is first-token KL against the original model over 100 prompts from mlabonne/harmless_alpaca. Thinking off. It is not a safety evaluation and says nothing about what the model will or will not produce.
Provided quants
| Quant | Size | PPL | KLD vs BF16 | Same top token | Notes |
|---|---|---|---|---|---|
| BF16 | 241.5 GB | 4.8665 | 0 (reference) | 100% | full precision, the reference for the KLD column |
| Q8_0 | 128.5 GB | 4.8688 | 0.0021 | 98.3% | fast, best quality |
| Q6_K | 112.9 GB | 4.8804 | 0.0054 | 97.0% | very good quality |
| Q5_K_M | 95.7 GB | 4.9244 | 0.0128 | 95.4% | |
| Q4_K_M | 86.1 GB | 4.9515 | 0.0340 | 92.6% | fast, recommended |
| Q4_0 | 71.8 GB | 4.9556 | 0.0336 | 93.4% | ggml-org mixture (see recipe) |
PPL and KLD: wikitext-2 test set, 100 chunks of 512 tokens, every quant scored against the logits of our BF16 GGUF. Same top token is how often the quant's most likely next token matches BF16's. Quants over 45 GB are split with llama-gguf-split into a folder of shards; point llama.cpp at the first shard (or use -hf) and it loads the rest.
Sidecars (top level, shared by every quant): mmproj-*.gguf is the vision projector (BF16, Q8_0) and mtp-*.gguf is NVIDIA's multi-token-prediction head as a separate draft model (BF16, Q8_0, Q4_0). With -hf, llama.cpp downloads the mmproj automatically and the MTP head when --spec-type draft-mtp is set, each at the precision closest to the quant.
Loading (any current llama.cpp)
llama-server -hf spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-GGUF:Q4_K_M --spec-type draft-mtp -ngl 99 -fa on -c 32768 --jinja
# local files: the quant (first shard if split) + sidecars
llama-server -m <QUANT>/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara.<QUANT>-00001-of-0000N.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf \
-md mtp-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf --spec-type draft-mtp -ngl 99 -fa on -c 32768 --jinja
Thinking: on by default in the template; pass chat_template_kwargs: {"enable_thinking": false} per request to turn it off.
Quant recipe
Converted from the full Omni checkpoint with llama.cpp master (convert_hf_to_gguf.py, main model with --no-mtp, plus --mmproj and --mtp for the sidecars). Q8_0 is converted straight from the BF16 safetensors; every other quant is cut from the BF16 GGUF, never requantized from another quant. K-quants use llama-quantize defaults. Q4_0 follows ggml-org's mixture: --pure Q4_0 with token_embd Q8_0, output Q6_K, and the shared experts, attention and Mamba (ssm_) tensors at Q8_0. The MTP Q4_0 is --pure.
License
OpenMDW 1.1, inherited from nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.