Ternary Nemotron 30B-A3B — 368K Context on 12 GB GPU (GGUF)

A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7 000+ tok/s prefill — in both stock llama.cpp and the PrismML fork. One file, both engines.

What this is

Nota AI ternarized NVIDIA's Nemotron 3.5 Lightning 30B-A3B to 1.58-bit weights, but published it in MLX format only (Apple Silicon). This repo re-packages those same weights — bit for bit, zero numerical loss — into standard GGUF Q2_0, so the model runs in llama.cpp and compatible engines on Linux / Windows / NVIDIA GPUs.

This is an intermediate recovery checkpoint (~89 % of the BF16 teacher's quality): excellent at reading, retrieval, summarisation and long-context comprehension; not a frontier coding model. See Quality & Limitations.

How it was made

The Nota checkpoint stores every linear weight as a true ternary matrix ({−s, 0, +s} per group of 64 inputs) in MLX's 2-bit affine container. GGUF Q2_0 (block 64, fp16 delta, codes {0,1,2} → {−d, 0, +d}) is bit-compatible with that format:

Verified: dequantizing the written Q2_0 blocks reproduces the MLX dequantization bit-identically across all 164 quantized tensors. Conversion was done with a custom offline tool (not distributed); the method above is everything needed to reproduce it.

Running

# stock llama.cpp (any 2026 build with nemotron_h_moe, validated on b11555+/b11796)
llama-server -m Ternary-Nemotron-3.5-Lightning-30B-A3B-GGUF.gguf \
    -c 262144 -np 1 -ngl 99 -fa on -ub 2048 \
    --cache-type-k q8_0 --cache-type-v q8_0 --port 8080

# PrismML fork (Bonsai engine, b10793) — same file, same flags
D:/Bonsai/bin/llama-server.exe -m Ternary-Nemotron-3.5-Lightning-30B-A3B-GGUF.gguf \
    -c 262144 -np 1 -ngl 99 -fa on -ub 2048 \
    --cache-type-k q8_0 --cache-type-v q8_0 --port 8080

Tuning guide (from extensive on-device validation)

Windows WDDM demotion warning: if total VRAM demand exceeds your card's dedicated memory, Windows silently moves ~1 GB to shared system memory → throughput drops ~20× (4 600 → 200 tok/s). Keep weights + KV + compute buffers under ~11.7 GB on a 12 GB card. Use -ngl 43 -ub 512 if you need extra headroom.

Do NOT use --kv-vram-cells (tiered KV, PrismML fork): it silently corrupts attention retrieval for nemotron_h_moe — verified: 0/3 needle with it, 3/3 without.

Reasoning model tip: pass chat_template_kwargs: {"reasoning_effort": "low"} (or use /completion instead of /v1/chat/completions) when you need short answers — otherwise the model may spend thousands of tokens thinking.

Benchmarks (RTX 4070 Ti 12 GB, all weights on GPU, q8_0 KV, flash-attn on)

Long-context needle retrieval (3 facts at 10/50/90 % depth, single pass)

Practical window: ~350–400K usable, 262K guaranteed.

Coding

This ternary model produces correct algorithms (monotonic stack, LFU with frequency buckets) but with subtle implementation bugs (off-by-one in the final drain, wrong eviction tie-break). This is consistent with the 89 % recovery checkpoint.

Negative experiment: embedding graft (transparency)

We grafted the original BF16 embeddings from the NVIDIA base into this GGUF (replacing the 4-bit-quantized ones from the Nota checkpoint). Pre-check confirmed the embeddings are frozen (zero training drift, pure quant noise). Result: coding scores identical to baseline — the deficit is architectural, not noise. The graft file is not published.

Quality & Limitations

This is Nota AI's intermediate ternary recovery checkpoint (published 2026-10-12; their "further recovered model" is available from the authors on request). Reported scores vs the BF16 teacher: GSM8K 95.00→89.31, MATH500 97.60→81.60, HumanEval 96.34→90.85, IFEval 92.05→83.36, HRM8K 91.05→83.24 (average 83.82 vs 93.80).

Strengths: fast, cheap, long-context reading and retrieval (172 tok/s on a 12 GB card, 368K working context). Good for summarisation, RAG over large documents, document QA, knowledge extraction.

Weaknesses: complex coding (subtle bugs under clear specs), complex mathematical reasoning (MATH500 −16 points), instruction-following precision (IFEval −9 points).

Known quirks:

• Runaway reasoning on hard prompts beyond its ability — use reasoning_effort: low or /completion.
• Tokenizer tagged pre=pixtral (same as all current Nemotron GGUFs, see llama.cpp #30894).
• The model has NO MTP (multi-token prediction) layer — the Nota checkpoint doesn't include one. Speculative decoding via --spec-type ngram-* works but gives no speedup at 172 tok/s baseline.

Engine compatibility matrix

Lineage

License

OpenMDW-1.1 (same as base and source checkpoint; see LICENSE). Commercial use is permitted. Derivatives must retain a copy of the agreement and the notices of origin. Attribution: base model © NVIDIA CORPORATION & AFFILIATES; ternary checkpoint by Nota AI; this GGUF is a lossless re-containerization produced by boro64.