Nemotron-Labs-3-Puzzle-75B-A9B — APEX GGUF quants

APEX (per-tensor, imatrix-guided precision allocation) quants of NVIDIA's Nemotron-Labs-3-Puzzle-75B-A9B, a hybrid Mamba2 / Attention / MoE model produced by NVIDIA's Puzzle neural architecture search, distilled down from Nemotron-3-Super-120B-A12B (120.7B/12.8B-active) to 75.3B total / 9.3B active parameters.

Requires a patched llama.cpp

nemotron_h_puzzle does not exist in mainline llama.cpp. Puzzle's MoE layers are genuinely heterogeneous per-layer — different expert widths (1280–2688) and different top-k values (4–18) from one MoE layer to the next — which mainline's nemotron_h_moe architecture doesn't support (it assumes one uniform expert width and top-k for the whole model).

A patch adding real per-layer support (arrays instead of scalars for expert width and top-k, threaded through both tensor creation and the MoE aggregation step) is included: brywil/llama.cpp, puzzle-nemotron branch (based on mainline llama.cpp, MIT license). The patch was verified bit-exact against the source model (every tensor, across every block type — mamba, MoE, attention — diffed to the original checkpoint with zero deviation, modulo the standard RoPE Q/K permutation every llama.cpp converter applies) before any quantization was done.

Build it:

git clone -b puzzle-nemotron https://github.com/brywil/llama.cpp llama.cpp-puzzle
cd llama.cpp-puzzle && cmake -B build -DGGML_CUDA=ON && cmake --build build -j
./build/bin/llama-server -m Nemotron-Labs-3-Puzzle-75B-A9B-APEX-i-quality.gguf --ctx-size 32768

Verified at 8167a021e, rebased onto mainline 6a32c29a7. Stock llama.cpp will refuse these files at the architecture check — every llama-server, llama-quantize and convert_hf_to_gguf.py invocation on this page means the binary from that build, which is why the path is written out rather than assumed to be on your PATH.

Convert + quantize with the patched build (mainline convert_hf_to_gguf.py + llama-quantize both work once patched — no special flags beyond the usual --imatrix/--tensor-type-file).

Quants

"Coherent" = real-generation check on a fixed prompt suite (not a canned demo). "Tool calls" = a 3-turn dependent tool-calling chain (with a distractor tool correctly never invoked), 3/3 trials succeeding at every tier.

Perplexity (wikitext, wiki.test.raw, 588 chunks, n_ctx=512 — identical methodology across every row below):

At matched tier names, Puzzle is measurably higher perplexity than the base model it was distilled from (worse quality) while being roughly 42% smaller — a real, quantified quality-for-size tradeoff from NVIDIA's Puzzle NAS compression, not a free lunch. Coherence and tool-calling both still hold at every tier (see above).

Separately, this model was run through a 5-task hard-tier coding benchmark (exact-context-window agentic tasks via OpenCode: a buddy allocator, a batch scheduler, a circuit breaker, an advanced expression parser, and an existing-codebase bug-fix/extend task) at the i-quality tier: 5/5 tasks fully passed, 59/59 individual test cases, 100% mean score — matching a same-size-class dense model (Qwen3.6-35B-A3B) exactly on the same suite, albeit taking substantially longer in wall-clock time (see the speed note below).

One structural note that affects all three tiers equally: 7 of 763 tensors (the ffn_down_exps weight on the 7 layers whose expert width is 2688 — not divisible by 256) fall back from the target quant type to a compatible alternative, since that width doesn't satisfy the block-size requirement of some low-bit formats. This is a real, minor, disclosed size cost (roughly 1-3GB depending on tier), not a defect.

Known limitation: generation speed

Measured decode speed on this quant is roughly 6.5–9 tokens/second, notably slower than the base Nemotron-3-Super model (~15-17 tok/s on mainline, uniform-MoE nemotron_h_moe) despite Puzzle having fewer total and active parameters. Root cause: ggml's CUDA-graph capture — which normally cuts per-token kernel-launch overhead substantially during decode — is disabled by upstream ggml whenever a MoE layer's per-token expert count exceeds a per-quant-type threshold (ggml_cuda_graph_check_compability in ggml-cuda.cu; see the TODO in that function and linked upstream PR discussion). Puzzle's per-layer top-k reaches 18 on some layers, well past these thresholds for several of the quant types used here; Super's uniform, smaller top-k stays under them throughout. This is an actively-tracked upstream ggml limitation, not something introduced by this patch, and may improve as that work lands.

Calibration

Nemotron-Labs-3-Puzzle-75B-A9B.imatrix (included in this repo) — 126-chunk calibration run over a diverse corpus (chat, code, reasoning, tool-calling), 392 entries, covering every block type (mamba/attention/MoE) at every distinct per-layer shape.

Not supported

MTP (multi-token prediction) draft head is dropped during conversion, same as every other Nemotron-H variant in mainline llama.cpp — no inference support for it exists upstream.