FP8 dynamic quantization of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated, produced for inference on NVIDIA DGX Spark (GB10, SM121) under its 128 GB unified-memory budget.
This is the clean abliteration variant quantized to FP8, distinct from existing Claude-distilled variants on the Hub. Uses llm-compressor FP8_DYNAMIC scheme + compressed-tensors float-quantized format.
⚠️ Vision pipeline fix (2026-05-03)
Pre-2026-05-03 uploads of this checkpoint had broken vision input — the model would emit
!!!!!token loops on any image. Root cause: vLLM (≤0.20.0) expects vision tensor keys atvisual.*, butQwen3_5MoeForConditionalGenerationsaves them undermodel.language_model.visual.*, so all 333 vision-tower tensors got silently skipped during load.This is now fixed: shard 19 +
model.safetensors.index.jsonre-uploaded with the visual prefix stripped. Text-only output is unaffected.If you cloned the model before 2026-05-03, the cleanest path is
huggingface_hub.snapshot_download(..., force_download=True, allow_patterns=["model-00019-of-00019.safetensors", "model.safetensors.index.json"])— only ~600 MB of changed weights.Diagnosis + remap script: see
remap_visual_path.pyin this repo.
| Metric | Value |
|---|---|
| Base | huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (BF16, 67 GB) |
| Format | compressed-tensors float-quantized (FP8 e4m3fn weights, dynamic per-token activation) |
| Disk size | ~36 GB (1.86× shrink from BF16) |
| Quant scheme | FP8_DYNAMIC via llm-compressor 0.10 |
| Quantized layers | 31,030 Linear modules → FP8 e4m3fn |
| Kept BF16 | lm_head, visual.*, mlp.gate (router), shared_expert_gate |
| MTP weights | Preserved (model_mtp.safetensors) for speculative decoding |
| Modality | Text + Image (vision tower preserved BF16, see fix note above) |
Single-stream decode, 200 tokens, GB10 (1× SM121, 128 GB UMA), vllm-node-tf5 build:
| Configuration | tok/s | vs BF16 |
|---|---|---|
| BF16 abliterated (this base) | 30.71 | 1.00× |
| This FP8 + MTP speculative | 51.72 | 1.68× |
Speculative decoding (qwen3_next_mtp, num_speculative_tokens=2) is the dominant contribution.
vllm serve coolthor/Huihui-Qwen3.6-35B-A3B-abliterated-FP8-DYNAMIC \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--reasoning-parser qwen3 \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Notes:
--kv-cache-dtype fp8 on GB10 — FP8 KV cache has known accuracy/repetition issues on SM121.--language-model-only if you want to feed images).For datacenter Blackwell (B200, RTX PRO 6000 with tensor-parallel-size > 1), follow the official Qwen 3.6 FP8 model card recipe and replace the model path. This artifact's FP8 layout is standard compressed-tensors, so it loads on any vLLM build that supports the architecture.
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_PATH = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"
SAVE_DIR = "Huihui-Qwen3.6-35B-A3B-abliterated-FP8-DYNAMIC"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
MODEL_PATH, dtype="auto", low_cpu_mem_usage=True,
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=[
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*shared_expert_gate$",
],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR, max_shard_size="2GB", safe_serialization=True)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_PATH, dest_dir=SAVE_DIR)
Qwen3_5MoeForConditionalGeneration writes a triple language_model. prefix in keys that vLLM's loader can't match (looks for model.language_model.). Strip one extra language_model.language_model. substring from each key in the saved safetensors before serving. (This is the LM-body fix; the visual tower needs an additional strip — see #2.)
Vision tower keys need an additional prefix strip. Qwen3_5MoeForConditionalGeneration writes vision weights as model.language_model.visual.*, but vLLM's qwen3_5.py loader looks for visual.* (no prefix). All 333 vision-tower tensors will silently skip-load and any image input produces !!!!! token loops with text-only output unaffected. Strip the model.language_model. prefix from every key matching model.language_model.visual.* in both the relevant safetensors shard and model.safetensors.index.json. The full remap script is at remap_visual_path.py in this repo (~50 lines, ~2 sec runtime — only one shard contains visual tensors, the other 18 can be hard-linked unchanged).
128 GB UMA is tight for save_pretrained — max_shard_size="2GB" is required; the default 50 GB shard buffer + GPU pool overlap will OOM-kill on Spark.
The full debugging journey (4 OOMs across 4 attempts + the vision-prefix fix) is documented in the accompanying blog post: Self-quantizing 35B abliterated MoE to FP8 on DGX Spark.
After the 2026-05-03 re-upload, vision input should work end-to-end. Quick smoke test:
import base64, json, urllib.request
with open("test.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
payload = {
"model": "qwen36-abliterated",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Describe this image in one sentence."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
]}],
"max_tokens": 200,
}
req = urllib.request.Request(
"http://localhost:8000/v1/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
print(json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"])
A working response returns a coherent description; a !!!!! loop means you're still hitting the prefix issue (your local copy is pre-fix — re-pull shard 19 + index.json).
Smoke-tested with prompts vanilla Qwen typically softens with disclaimers (villain monologue, technical security topics, dark humor, persona requests). All five test prompts returned direct content with zero refusal markers. Identity-level "are you uncensored?" returns the standard Qwen self-description (training-data residue), but functional refusal behavior is suppressed.
Inherits from base model. The original abliteration is Apache-2.0 (Huihui's remove-refusals-with-transformers derivative). Qwen 3.6 base is Apache-2.0.
⚠ Safety: This model has reduced refusal behavior. Use responsibly.
model-00019-of-00019.safetensors and model.safetensors.index.json with visual.* prefix corrected; added remap_visual_path.py and updated docs. No change to text-only behavior or benchmarks. Earlier downloads silently skipped 333 vision-tower tensors on load.☕ If this saved you GPU hours, you can buy me a coffee.
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
This section documents release fe4d199c989e78d923dad976308ecdb5bd173771, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
| Source repository | Revision used for this build | Role |
|---|---|---|
| huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated | not recorded | Immediate BF16 input; upstream already abliterated |
Tools and versions: llm-compressor 0.10 (version granularity in existing card); compressed-tensors exact version and converter commit not recorded.
The model was quantized with FP8_DYNAMIC, with the LM head, vision tower and MoE routing gates left unquantized. It was saved in 2 GB shards with MTP tensors copied. The 2026-05-03 vision-prefix repair renames model.language_model.visual.* to visual.* and must be included when reproducing this published version; it is a key remap rather than fine-tuning.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Build script (copied from the existing card, not re-run): Because it references the repo by name without a pinned commit, the exact upstream commit used at build time is unknown.
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_PATH = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"
SAVE_DIR = "Huihui-Qwen3.6-35B-A3B-abliterated-FP8-DYNAMIC"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
MODEL_PATH, dtype="auto", low_cpu_mem_usage=True,
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=[
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*shared_expert_gate$",
],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR, max_shard_size="2GB", safe_serialization=True)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_PATH, dest_dir=SAVE_DIR)
The serialized quantization settings are in recipe.yaml.
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
model-00001-of-00019.safetensors |
1017118816 | 6abf5b16b9189c4e53a99c548a1317990c48edec364b0fc4db016850c0863993 |
model-00002-of-00019.safetensors |
2000038608 | 29ec5f7eb334d3707ad0605337ce3d8bda28dbf80f6491da626d00cfb5081b62 |
model-00003-of-00019.safetensors |
2000037696 | 44fc9696dc0bcd62d9a401f788363f4017c27f13e24d23e97a96f471108ec1ff |
model-00004-of-00019.safetensors |
2000156968 | d9b3313f8197c831872dd9e17a5219bad565fcfa95c1e4f4b42b763498567144 |
model-00005-of-00019.safetensors |
2000165448 | 922c592017c973b74ae326e406fd4428ea58bcc10d209824845329a38ed418e0 |
model-00006-of-00019.safetensors |
2000194296 | c04df652ecb1d690f6fdc2826513fb7bf30195d5baedce048efd45c0be0e1d19 |
model-00007-of-00019.safetensors |
2000192096 | 0f849d16f314f7c8bf8a0b511b236a42d03d22083c4443f5631123a15862a432 |
model-00008-of-00019.safetensors |
2000018248 | f980288ae6756532be83470a719bfa8c8768c148be56010ddccbdd9a07c3215b |
model-00009-of-00019.safetensors |
2000160392 | 6600d542b806e3c33b72ec28ecdb568cf9987de0a1bb3040e46eb783a2ca7499 |
model-00010-of-00019.safetensors |
2000169104 | 0953b1f5ff8b70b3edaecdfc05338e1a591ad53cf7e0ba1435f87f0cfeb458da |
model-00011-of-00019.safetensors |
2000193496 | 1b8f356808e8faa2ebdf96f876dd09164cb70622c8936decc906f434945a7c56 |
model-00012-of-00019.safetensors |
2000008336 | f27e287d0c5dc7bc62d1b027767be66f91f1791e8335cc0c80e09e3acff8c488 |
model-00013-of-00019.safetensors |
2000205832 | 26f86a541ccfe1fcd0497f78a2c982d280b7010bc569f7deabfa804f8971d585 |
model-00014-of-00019.safetensors |
2000157056 | 9c401214fcaae6f169dfdab59b952b80abe7010d848996fe437312a2b358095b |
model-00015-of-00019.safetensors |
1988474480 | dbb28278768b5c0478d4af98ce3df9cc3a8a8c173d3811c1f7f2f59ee7f051e6 |
model-00016-of-00019.safetensors |
2000330448 | f8287554ce172e93720d781f6d2c96d4f8f564f6ec415f2f7c3c967c6cb51067 |
model-00017-of-00019.safetensors |
2000008040 | 27ec8b5a29d64f7b000f48cd4b5c7e38901d2a8f29af28ee925f942bd51506ea |
model-00018-of-00019.safetensors |
2000205776 | 35c253a088da5a1ef6709301e85a40e2888b5b93e9b1443156189629c23885d0 |
model-00019-of-00019.safetensors |
1658207312 | 95eafd28fc39444bba643d6a8f356629555a21d313ab7eb19cc381f35b04d526 |
model_mtp.safetensors |
1689283656 | 7d0030f5dda6c9f2968b91ca835e0de97b3090f003c5acab4e71aad029601410 |
Download the desired files from revision fe4d199c989e78d923dad976308ecdb5bd173771 and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Base revision; Exact tool patch versions/commit; Fully pinned build-time execution environment.
Sources: existing model card and HF metadata.
model-00016-of-00019.safetensors 2.0 GBmodel-00013-of-00019.safetensors 2.0 GBmodel-00018-of-00019.safetensors 2.0 GBmodel-00006-of-00019.safetensors 2.0 GBmodel-00011-of-00019.safetensors 2.0 GBmodel-00007-of-00019.safetensors 2.0 GBmodel-00010-of-00019.safetensors 2.0 GBmodel-00005-of-00019.safetensors 2.0 GBmodel-00009-of-00019.safetensors 2.0 GBmodel-00014-of-00019.safetensors 2.0 GBmodel-00004-of-00019.safetensors 2.0 GBmodel-00002-of-00019.safetensors 2.0 GBmodel-00003-of-00019.safetensors 2.0 GBmodel-00008-of-00019.safetensors 2.0 GBmodel-00012-of-00019.safetensors 2.0 GBmodel-00017-of-00019.safetensors 2.0 GBmodel-00015-of-00019.safetensors 2.0 GBmodel_mtp.safetensors 1.7 GBmodel-00019-of-00019.safetensors 1.7 GBmodel-00001-of-00019.safetensors 1.0 GB