NVFP4 W4A4 quantization (4-bit weights and 4-bit activations) of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated — an uncensored / abliterated Qwen3.6-35B-A3B hybrid MoE (~3B active, GDN linear-attention + full-attention interleave). Self-quantized with llm-compressor (NVFP4 scheme) in compressed-tensors nvfp4-pack-quantized format.
The point of this checkpoint: on a single NVIDIA DGX Spark (GB10, SM121) it is servable in vLLM and decodes faster than the FP8 build — which most NVFP4 MoE quants on consumer/SM121 Blackwell are not (they typically fall back to a dequant slow path or refuse to compile the block-scaled MMA). With the right toolchain it lands at 66.9 tok/s vs 52.0 tok/s for FP8 (+29%) while using 16 GB less weight memory.
Lineage: BF16 abliterated base (huihui-ai) → NVFP4 W4A4 (this repo). Vision tower preserved and the model.language_model.visual.* → visual.* loader-prefix fix already applied (see note below). MTP heads preserved but off by default (see below).
| Metric | Value |
|---|---|
| Base | huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (BF16) |
| Format | compressed-tensors nvfp4-pack-quantized |
| Scheme | NVFP4 — W4A4 (weights FP4 + activations FP4), group size 16, FP8 e4m3fn scales |
| KV cache scheme | fp8 |
| Disk size | ~22 GB |
| Kept higher precision | lm_head, visual.*, mlp.gate (router), shared_expert_gate |
| MTP weights | Preserved (model_mtp.safetensors) — keep OFF, see note |
| Native context | 256K (max_position_embeddings 262144), no YaRN |
| Modality | Text + Image (vision tower preserved, loader-prefix fixed) |
Paired, same-harness, single-stream decode, KV-cache fp8, warm:
| Configuration | Weight mem | tok/s | vs FP8 |
|---|---|---|---|
| FP8 dynamic | ~38 GB | 52.0 | 1.00× |
| This NVFP4 W4A4 (cudagraph on) | ~22 GB | 66.9 | 1.29× |
This NVFP4 W4A4, --enforce-eager |
~22 GB | ~23 | 0.44× |
The whole speedup lives in CUDA graphs. With
--enforce-eagerthis model is only ~23 tok/s. On a hybrid MoE, eager-mode launch overhead from the many small per-expert kernels dominates wall-clock; capturing CUDA graphs removes it and recovers the full 66.9 tok/s. Do not pass--enforce-eager.
Full write-up (method, sweep, traces): DGX Spark Part 33 — NVFP4 W4A4 MoE + CUDA graphs.
Toolchain requirement. You need a vLLM build where the SM121 NVFP4 block-scaled MMA actually compiles: cutlass-dsl 4.5.x + flashinfer 0.6.11+. Do not use any sm121-b12x style mod that downgrades cutlass-dsl to 4.4.2 — that breaks the compiled NVFP4 path and silently forces enforce-eager behaviour (back to ~23 tok/s).
vllm serve coolthor/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4 \
--served-model-name qwen36-abliterated \
--moe-backend flashinfer_cutlass \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.50 \
--max-model-len 262144 \
--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--enable-prefix-caching \
--trust-remote-code
--enforce-eager — it drops you from 66.9 → ~23 tok/s (kills the CUDA-graph win).--max-num-batched-tokens >= 2096 is required — the hybrid Mamba/GDN cache block alignment needs it; smaller values fail to allocate the conv/state cache.flashinfer_cutlass NVFP4 MoE backend wants W4A4 (this model). A W4A16 weight-only NVFP4 export is rejected by this backend — the activations must be FP4 too, which is why this checkpoint is W4A4.num_speculative_tokens n=0–4; the no-spec baseline wins. On this hybrid at GB10 memory bandwidth, NVFP4 weight loading and MTP speculation are substitutes competing for the same bandwidth — stacking them does not help. Just don't pass a --speculative-config.Vision loader-prefix fix (carried forward)
Qwen3_5MoeForConditionalGenerationsaves vision weights undermodel.language_model.visual.*, but vLLM's loader looks forvisual.*, so all vision tensors silently skip-load and image input degenerates into!!!!!loops (text-only unaffected). This checkpoint already has the prefix stripped in the affected shard +model.safetensors.index.json, so vision works out of the box.
# recipe.yaml
default_stage:
default_modifiers:
QuantizationModifier:
targets: [Linear]
ignore: ['re:.*lm_head', 're:visual.*', 're:model.visual.*', 're:.*mlp.gate$', 're:.*shared_expert_gate$']
scheme: NVFP4
bypass_divisibility_checks: false
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_PATH = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"
SAVE_DIR = "Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
MODEL_PATH, dtype="auto", low_cpu_mem_usage=True,
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
recipe = QuantizationModifier(
targets="Linear",
scheme="NVFP4", # W4A4: 4-bit weights + 4-bit activations
ignore=[
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*shared_expert_gate$",
],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR, max_shard_size="2GB", safe_serialization=True)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_PATH, dest_dir=SAVE_DIR)
Apache-2.0, inherited from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated and the upstream Qwen/Qwen3.6-35B-A3B (Apache-2.0). Redistribution permitted.
☕ If this saved you GPU hours, you can buy me a coffee.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
This section documents release 6b057bce5a4965c9bb661bb6b2fb2e648d806984, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
| Source repository | Revision used for this build | Role |
|---|---|---|
| huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated | not recorded | Immediate BF16 input; upstream already abliterated |
Tools and versions: llm-compressor and compressed-tensors; exact build-time versions/commits not recorded.
The model was quantized to NVFP4 (W4A4, compressed-tensors nvfp4-pack-quantized) with the LM head, vision tower and MoE routing gates left unquantized, and saved in 2 GB shards with MTP tensors preserved. The card includes the oneshot script but not the exact calibration data or run logs.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Build script (copied from the existing card, not re-run): Because it references the repo by name without a pinned commit, the exact upstream commit used at build time is unknown.
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_PATH = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"
SAVE_DIR = "Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
MODEL_PATH, dtype="auto", low_cpu_mem_usage=True,
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
recipe = QuantizationModifier(
targets="Linear",
scheme="NVFP4", # W4A4: 4-bit weights + 4-bit activations
ignore=[
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*shared_expert_gate$",
],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR, max_shard_size="2GB", safe_serialization=True)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_PATH, dest_dir=SAVE_DIR)
The serialized quantization settings are in recipe.yaml.
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
model-00001-of-00012.safetensors |
1017118816 | 6abf5b16b9189c4e53a99c548a1317990c48edec364b0fc4db016850c0863993 |
model-00002-of-00012.safetensors |
2000842472 | f010a80d8237c587d2193d78ad556b08bc7b08b215961e909d9568ef14f8335d |
model-00003-of-00012.safetensors |
2001812200 | 6df4c8c2bc8d5784474dc98f619ff4f3af8d59c36d78a7fc73640cff3a5fba5f |
model-00004-of-00012.safetensors |
2001355728 | 7fae6c901032b8172f15af5deb51adfba09969530acc9e509eeca5eb319c03ba |
model-00005-of-00012.safetensors |
2001826272 | 9df5cff9e1a829af5ff115e947834f1b720109b772ffcfa9634a9e414a4e2a02 |
model-00006-of-00012.safetensors |
2001367792 | b73c2d2b757ea3dc8f885ee9b4f4d9846153d61447933508058b7c5edce837b8 |
model-00007-of-00012.safetensors |
2001772232 | f926fff8560b12bc55523c48d8b0b724a73f10559ed688a65410d50db0a2246e |
model-00008-of-00012.safetensors |
2001825624 | c4c3cb40163541803bbc19207a0f9ca87627089573189e2919d6f575d8b8cc7f |
model-00009-of-00012.safetensors |
2001367904 | e02dc39b6286f58d7ff4555970ac5aee53024ae0d06fb60d91232b91a1c02dce |
model-00010-of-00012.safetensors |
2001826376 | 9bf01bdbb2d407492d0c71955ccf37b465f9f6772023f065c8b6c5bd3820ca43 |
model-00011-of-00012.safetensors |
1997585688 | 76bb77c78ea805f7b813f9614710eb24c6fdf82aa3a79e84416b4bbcd4446733 |
model-00012-of-00012.safetensors |
872623384 | 8fb51895b78f0e8aeec1363649875ba9eff85f97c379703b5416d877be27938e |
model_mtp.safetensors |
1689283656 | 7d0030f5dda6c9f2968b91ca835e0de97b3090f003c5acab4e71aad029601410 |
Download the desired files from revision 6b057bce5a4965c9bb661bb6b2fb2e648d806984 and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Base revision; Exact tool versions/commits; Fully pinned calibration and execution record.
Sources: existing model card and HF metadata.
model-00010-of-00012.safetensors 2.0 GBmodel-00005-of-00012.safetensors 2.0 GBmodel-00008-of-00012.safetensors 2.0 GBmodel-00003-of-00012.safetensors 2.0 GBmodel-00007-of-00012.safetensors 2.0 GBmodel-00009-of-00012.safetensors 2.0 GBmodel-00006-of-00012.safetensors 2.0 GBmodel-00004-of-00012.safetensors 2.0 GBmodel-00002-of-00012.safetensors 2.0 GBmodel-00011-of-00012.safetensors 2.0 GBmodel_mtp.safetensors 1.7 GBmodel-00001-of-00012.safetensors 1.0 GBmodel-00012-of-00012.safetensors 873 MBhuihui-ai/Huihui-Qwen3.6-35B-A3B-abliteratedHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo coolthor/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4-W4A4 ./model-folder