Dory1.0-4B GGUF (imatrix quants)

GGUF quantizations of nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16, with KL-divergence measurements for every file at 1, 2 and 4 loops.

This is an independent, unofficial quantization; it is not affiliated with or endorsed by NVIDIA. NVIDIA's own GGUF is a single Q5_K_M at nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF.

TL;DR

• Pick Q4_K_M (2.43 GB) as the default, Q5_K_M (2.84 GB) or Q6_K (3.27 GB) if you have room, IQ3_M (1.92 GB) if you don't.
• Our Q5_K_M is 23% smaller than NVIDIA's Q5_K_M with the same KLD, and decodes ~2.8× faster on CUDA (see Differences from NVIDIA's GGUF).
• Needs a llama.cpp build with Dory support (ggml-org/llama.cpp#30306, not merged yet).
• The GGUFs default to 4 loops; NVIDIA recommends starting at 2. Set it per run with --override-kv (below).

The base model

Dory1.0-4B is a looped (latent-recurrent) 4B LLM from NVIDIA's Nemotron Labs. A shared middle block of layers is applied repeatedly per token, so one checkpoint covers several compute budgets: more loops means more compute per token and higher accuracy, without changing the weights.

Benchmarks, intended use and training data: see NVIDIA's model card.

Running it

1. Build llama.cpp with Dory support

Dory support is in ggml-org/llama.cpp#30306 (unmerged). These files were made and tested at PR revision 43e865ec059a35d57f1167eaea76ca187077f063. CUDA example:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/30306/head
git checkout 43e865ec059a35d57f1167eaea76ca187077f063   # or FETCH_HEAD for the latest PR head
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-cli

Use -DGGML_METAL=ON on Apple Silicon or leave GPU flags off for CPU. You don't need the patch in llama.cpp/ to run these files, only to reproduce the conversion.

2. Download a quant

hf download cloudnathan5/Dory1.0-4B-GGUF Dory1.0-4B-Q4_K_M.gguf --local-dir .

3. Serve

./build/bin/llama-server -m Dory1.0-4B-Q4_K_M.gguf \
    -ngl 999 -c 32768 --jinja --reasoning on \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 \
    --override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"

Web UI at http://127.0.0.1:8080, OpenAI-compatible API at /v1. Sampling settings are NVIDIA's recommendation.

• Loops: dory.num_loops=int:1|2|3|4. Without the override the files run at 4.
• Use -ngl 999, not the layer count. llama.cpp counts virtual layers (attention and FFN separately, recurrent ones once per loop: 192 at 4 loops), so a smaller -ngl silently leaves part of the model on the CPU.
• Long context on small GPUs: dory.recurrent_kv_cache_mode=str:last_loop keeps KV at the 1-loop size at any loop count, and/or add -ctk q8_0 -ctv q8_0 (see the memory table below).

Which file?

Pareto-optimal set, KL divergence vs BF16 at 2 loops (lower is better; top-1 = how often the quant's most likely token matches BF16's):

The other stock quants (Q5_K_S, IQ4_NL, Q3_K_L/M/S, IQ3_S, Q2_K, Q2_K_S) and the BF16 GGUF are also in the repo root. They are dominated on KLD per byte but may run faster on backends where I-quant kernels are slow (CPU, some Metal paths). For quality at 2 bits, IQ2_M is the floor we'd consider usable.

Memory

The weights are small; the KV cache is what grows. In the default per_loop mode at 4 loops there are 24 full-attention layer instances (2 input + 5 recurrent × 4 loops + 2 output), each storing 4 KV heads × 256 dims for K and V: ~97 KiB per token with an F16 cache, ~55 KiB with q8_0. Sliding-window layers add a small fixed amount.

Measured total VRAM (nvidia-smi, llama-server, one slot, flash attention auto, RTX PRO 6000):

At 4K: IQ4_XS 3.4, Q4_K_M 3.6, Q5_K_M 3.9, Q6_K 4.3, Q8_0 5.1 GiB. Rough guide: an 8 GB card runs Q4_K_M/Q5_K_M at ~32K (F16 KV) or ~64K (q8_0 KV), more with last_loop; a 16 GB card runs Q8_0 at 64K, or Q4_K_M at 128K with q8_0 KV. Leave headroom if the GPU also drives a display.

Speed

Decode, batch 1, RTX PRO 6000 Blackwell (CUDA 13.3): Q5_K_M 193.5 tok/s at 1 loop, 80.9 tok/s at 4 loops. Cost per token scales roughly with the number of layer passes (36 at 1 loop, 96 at 4).

How these were made

• Source: BF16 safetensors → BF16 GGUF with the PR's converter, plus a one-line converter patch that stores the nGPT scale vectors as F32 (see Differences). All quants are made from that BF16 GGUF.
• imatrix (imatrix/imatrix-dory-4loop.gguf): ~927K tokens of Nemotron-Post-Training-Dataset-v1 rendered with Dory's chat template (chat, code, math, STEM, tool calling), computed at 4 loops so the recurrent block's statistics include every loop (it collects exactly 4× the samples of the input/output blocks). ctx 4096, 160 chunks.
• Recipes: llama.cpp's stock recipe for each type, with the imatrix. Nothing custom in the root files.
• Evaluation: llama-perplexity --kl-divergence vs the BF16 GGUF on 16 × 4096-token chunks (~32K scored tokens) of held-out data from the same dataset (disjoint from calibration), at 1, 2 and 4 loops. BF16 PPL on this text: 2.340 / 2.284 / 2.291 at 1 / 2 / 4 loops.

Findings

1. Quant error does not compound with loops. KLD at 4 loops ÷ KLD at 1 loop is 0.98-1.17 for everything Q5 and below, even though the recurrent weights run 4× per token. Only the near-lossless quants grow in relative terms (Q8_0 2.1×), and they stay tiny. nGPT's re-normalization onto the hypersphere after each block probably keeps errors from accumulating. One file serves every loop count.
2. The recurrent block isn't special, and embeddings/LM head precision barely matters. We tried _XL variants (recurrent block one quant level higher) and _L variants (Q8_0 embeddings + output). Every _XL was beaten by a smaller stock quant, and every _L was within noise of its base. They're in experimental/ for reference.
3. The imatrix matters a lot: Q4_K_M 0.030 vs 0.074 without it; Q2_K 0.18 vs 0.70 without it (the no-imatrix Q2_K is also the only file that degrades with loops).
4. I-quants dominate below 4 bits: IQ3_M (1.92 GB) beats Q3_K_L (2.14 GB); IQ2_M is the same size as Q2_K_S with lower KLD (0.19 vs 0.24); IQ4_NL is dominated by IQ4_XS.
5. NVFP4: NVIDIA's own NVFP4 checkpoint (ModelOpt, for vLLM) is Q4_K_M quality (KLD 0.032 at 2 loops, measured by dequantizing it exactly). llama.cpp's NVFP4 type is not: 0.111 quantized from BF16, and 0.098 even when repacking NVIDIA's NVFP4 weights into it. ggml's NVFP4 has no per-tensor scale, so NVIDIA's block scales can't be carried over and get re-derived by plain amax, and Dory's small weights push almost all of the FP8 block scales into the subnormal range. In llama.cpp, use Q4_K_M or IQ4_XS instead. Both ggml NVFP4 files are in experimental/.

Differences from NVIDIA's GGUF

NVIDIA's Q5_K_M uses llama.cpp's stock Q5_K_M layer mix, but keeps the embeddings and LM head in BF16 (1.34 GB of its 3.67 GB) and uses no imatrix. Compared with it, our Q5_K_M:

• is 23% smaller with the same quality: 2.84 GB vs 3.67 GB, KLD 0.0150 vs 0.0157 at 2 loops;
• is ~2.8× faster on CUDA. The PR's converter writes the nGPT scale vectors (*_scale, residual_alpha) as BF16 in BF16 exports. They're consumed by GGML_OP_MUL, which CUDA doesn't support for F32×BF16, so llama.cpp keeps them in host memory and runs a BF16→F32 cast on the CPU for every layer (435 graph splits per token). Storing them as F32 (+2 MB) brings it to 2 splits: Q5_K_M 69.7 → 193.5 tok/s at 1 loop, 27.5 → 80.9 tok/s at 4 loops. NVIDIA's file has the same issue. The fix is llama.cpp/llama.cpp-dory-f32-scales.patch (converter only; all files here already have F32 scales).

Our exact reproduction of NVIDIA's recipe (experimental/Dory1.0-4B-Q5_K_M-NVrecipe.gguf: BF16 embeddings/output, no imatrix, identical tensor types) scores 0.028, worse than NVIDIA's file (0.016). NVIDIA quantized from a pre-BF16 export, and that probably explains the gap.

Full results

KLD vs BF16 (mean) and top-1 agreement, at 1 / 2 / 4 loops. ⚗ = in experimental/; † = reference, not in this repo.

Raw numbers incl. 99th-percentile/max KLD and PPL: eval/kld_results.csv.

Experimental files

Reproducing

Scripts are in tools/:

1. Build the PR (above) and git apply llama.cpp-dory-f32-scales.patch in the llama.cpp tree.
2. tools/adapt_config.py <hf_dir> <out_dir>: the PR's converter expects NVIDIA's internal config keys, and this translates the public config.json (resulting GGUF metadata matches NVIDIA's). Then convert_hf_to_gguf.py <out_dir> --outtype bf16.
3. tools/build_calib.py: builds calibration and held-out text from Nemotron-Post-Training-Dataset-v1.
4. llama-imatrix ... --override-kv dory.num_loops=int:4 (or use the shipped imatrix).
5. tools/quantize_all.sh: the whole lineup.
6. tools/eval_kld.sh + tools/summarize_kld.py: KLD at 1/2/4 loops.
7. tools/dequant_modelopt_nvfp4.py: dequantizes NVIDIA's ModelOpt NVFP4 checkpoint for the comparison above.

Caveats

• Dory support in llama.cpp is an unmerged PR; the GGUF format for this architecture could still change before it lands.
• Calibration and evaluation text come from the same dataset, so the imatrix's advantage may be slightly overstated on out-of-domain text. No task benchmarks were run on the quants.
• Speed and memory numbers come from one GPU (RTX PRO 6000 Blackwell); expect similar memory and proportionally lower speed elsewhere.

License

The model weights are licensed by NVIDIA under the OpenMDW License Agreement, version 1.1. These quantizations are derivatives distributed under the same license. Base model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16.