Nanbeige4.3-33B-NVFP4-GGUF

HF repo: esatapedico/Nanbeige4.3-33B-NVFP4-GGUF (this file). Conversion checkpoint (2-shard NVFP4A16 safetensors): esatapedico/Nanbeige4.3-33B-NVFP4.

NVFP4 GGUF ladder of Nanbeige/Nanbeige4.3-33B (dense 33B Llama-arch, Apache-2.0): 52 layers, 7168 hidden, 56 query / 8 KV heads, 512K native context, no MoE/SWA/SSM/MTP. Converted by MLP-only NVFP4A16 self-conversion (weight-only, datafree, no calibration corpus).

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.

🌐 esatapedico.com. Updates, work-in-progress, and ways to support.

Disclaimer: read before downloading

These are self-converted, weight-only quantizations with no calibration corpus. MEDIUM was measured on our suite at 67.62% MMLU and 91.5% GSM8K (see Measurements). Scores move with runtime, KV-cache type, context length, and sampling settings, so validate on your own workload before depending on a tier in production.

At a glance

The files

The six NVFP4 tiers share one 364-tensor NVFP4 backbone (156 MLP + 208 attention), byte-identical across tiers (SHA-verified). HIGHEST keeps the 156-tensor MLP backbone and requants attention to Q8_0.

No tier fits a single 16 GB card (floor 16.94 GiB). KV cache: q4_0, 58.5 KiB/token (52/52 full-attention layers) → 32K 1.83 GiB, 64K 3.66, 128K 7.31. Validated for full offload on dual 16 GB Blackwell cards; any NVFP4-capable llama.cpp backend with enough VRAM works. q4_0 KV is our quality-neutral record (paired mainline test, see repo docs).

Measurements

bench_suite --tier gate --stages Q per tier, dual 16 GB Blackwell, server-default KV cache (no --ctk/--ctv flags — identical harness for all tiers, so the tiers are mutually comparable). These numbers are reported as information, not as a pass/fail bar: MMLU-1000 (seed 42, letter-logprob, thinking forced off) and GSM8K-200 (generative, thinking on). IFEval parsed 30/30 on every tier; GPQA skipped (no HF token).

Unresolved MMLU items (long prompts over the harness context, auto-scored 0) are excluded from the denominator on every tier. MEDIUM was additionally confirmed by an identical rerun with per-item identical answers. The six NVFP4 tiers share the byte-identical backbone listed above and differ only in embedding and output-head precision, and HIGHEST additionally requants attention to Q8_0.

Usage

MEDIUM at 128K on two offloaded cards (adjust -c per the ctx caps in the table above; HIGHEST example uses -c 65536):

llama-server \
  -m Nanbeige4.3-33B-NVFP4-MEDIUM.gguf \
  -c 131072 -ngl 99 --jinja \
  --cache-type-k q4_0 --cache-type-v q4_0

Check flag names against your build's --help before relying on them.

How this was made

1. BF16 source. Nanbeige/Nanbeige4.3-33B (14 shards, 66.3 GB, 471 tensors = 52×9 + 3, no MTP), SHA-256 verified 14/14 against HF LFS oids.
2. MLP-only NVFP4A16 self-conversion. llm-compressor 0.14.0.1, pipeline='sequential', weight-only, no calibration, targets = 156 MLP gate/up/down_proj (attention, norms, heads ignored in both namespaces; recipe validated against weight_map before running, dry-run green). Post-quant norm guard compared 315 untouched tensors: 0 mutated. Output: indexed 2-shard checkpoint (~30.8 GB) with recipe.yaml, resaved config, intact tokenizer.
3. ORIG GGUF. convert_hf_to_gguf.py --outtype auto (llama.cpp pin b3fa6995, maps LlamaForCausalLM → llama) → 627 tensors: 156 NVFP4 + 156 F32 scales + 208 BF16 attention + 104 F32 norms + output_norm + 2 BF16 heads. Gated with check_nvfp4 (--min-nvfp4-tensors 156, --expect-bf16-re for the intentional BF16 attention).
4. Tier builds. CPU-only llama-quantize (built GGML_CUDA=OFF from the pinned source; the image binary needs libcuda, absent on the build box) with per-tier override maps (627 entries each): six tiers pin the 364-tensor backbone to nvfp4, HIGHEST pins 208 attention tensors to q8_0.
5. Verification. Per-tensor SHA-256 across all seven tiers after the rebuild: ORIG's 156 NVFP4 MLP tensors are byte-identical in every tier, and the 364-tensor backbone (156 MLP + 208 attention) is byte-identical across the six NVFP4 tiers; each tier additionally passes check_nvfp4 (364 NVFP4 tensors in the six ladder tiers, 156 in HIGHEST).

Attribution & provenance

Derivative work built from Apache-2.0 artifacts. Nothing here was trained or fine-tuned. Credit belongs to:

1. Nanbeige for the base model this ladder converts.
2. llama.cpp contributors for the GGUF toolchain and the NVFP4 path with Blackwell kernels.
3. Red Hat AI / vLLM project for llm-compressor, used for the NVFP4 quantization.
4. esatapedico for the conversion, verification, and benchmarking.

"Nanbeige" is a name of the base model publisher. This repository is not affiliated with, sponsored by, or endorsed by the publisher.

License

Apache-2.0, identical to every upstream artifact (the base model ships Apache-2.0; no LICENSE file is bundled upstream, so this repo carries its own Apache-2.0 text with attribution). GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to the base model publisher as above.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the files themselves before relying on it.