Nemotron-3.5-Lightning-30B-A3B GGUF

GGUF quants of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 for llama.cpp.

Built and tested on:

Use a recent llama.cpp that includes Nemotron-H MoE (nemotron_h_moe). Older trees without that arch will not load these files.

Files

Benchmark (speed only)

Setup (this table): RTX 3090 24 GB + 64 GB RAM, llama.cpp llama-server, Q6_K,
-ngl 999 -ncmoe 22 -np 1 -fa on --jinja, large context (~200k slot).
No MTP / no speculative decoding for these numbers (--spec-type not used).
Measured with llama-benchy: pp4096 + tg256 at several context depths.

Numbers are single-run; your results will vary with build flags, drivers, -ncmoe, and load.

Quick start

llama-server (no MTP) — matches the bench style

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q6_K.gguf \
  -ngl 999 \
  -ncmoe 22 \
  -np 1 \
  -fa on \
  --jinja \
  -c 200000 \
  --alias nemotron-q6k \
  --host 0.0.0.0 --port 8080

Q4_K_M: same flags, swap the -m path.

llama-server with MTP (speculative multi-token prediction)

MTP weights are in the GGUF. Enable built-in draft-MTP (no separate draft model file):

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q6_K.gguf \
  -ngl 999 \
  -ncmoe 22 \
  -np 1 \
  -fa on \
  --jinja \
  -c 200000 \
  --alias nemotron-q6k \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --host 0.0.0.0 --port 8080

Tune --spec-draft-n-max (e.g. 2–4). Acceptance/speedup depends on workload; re-bench if you care about TG with MTP on.

llama-cli

llama-cli \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf \
  -ngl 999 \
  -ncmoe 16 \
  -c 8192 \
  -n 256

From this Hub repo

llama-server \
  --hf-repo aj9o9/nvidia-nemotron-3.5-lightning-30b \
  --hf-file NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q6_K.gguf \
  -ngl 999 -ncmoe 22 -np 1 -fa on --jinja -c 200000 \
  --alias nemotron-q6k

# with MTP
llama-server \
  --hf-repo aj9o9/nvidia-nemotron-3.5-lightning-30b \
  --hf-file NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q6_K.gguf \
  -ngl 999 -ncmoe 22 -np 1 -fa on --jinja -c 200000 \
  --alias nemotron-q6k \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

VRAM / OOM (prefer expert offload, not lower -ngl)

Keep -ngl 999 (full GPU layer offload). If you OOM, offload MoE expert weights to system RAM:

# examples
llama-server -m ...-Q6_K.gguf -ngl 999 -ncmoe 8  -c 200000
llama-server -m ...-Q6_K.gguf -ngl 999 -ncmoe 22 -c 200000   # used for the speed table
llama-server -m ...-Q6_K.gguf -ngl 999 -cmoe      -c 200000   # max expert offload

Raise -ncmoe until it fits. Only lower -c if you still OOM after expert offload. 64 GB system RAM is what makes -ncmoe practical on a 3090-class box.

Prompting

ChatML-style template (<|im_start|> / <|im_end|>) with optional thinking and tool-calling. Prefer chat mode or /v1/chat/completions so the embedded template is applied. Server may suggest --reasoning-preserve for this template.

Quality notes

• Straight K-quants from BF16 GGUF (no imatrix).
• Q4_K_M: smaller / faster to load; Q6_K: higher fidelity (bench table above).
• Small quality drop vs BF16/Q8; solid for chat and coding on 3090 + 64 GB RAM.

License

OpenMDW License Agreement, version 1.1 as with the base NVIDIA model materials.

• https://openmdw.ai/license/1-1/
• https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Retain the license and applicable notices when redistributing.

Credits

• Base model: NVIDIA
• Runtime: ggml-org/llama.cpp

Disclaimer

Not affiliated with NVIDIA. Provided as-is. Follow the base model card for intended use, safety, and limitations.