Qwen3-VL-8B-Thinking-heretic-1refusal — NVFP4 + ConvRot INT8

Quantized from Kizzington/Qwen3-VL-8B-Thinking-heretic-1refusal, itself a Heretic-decensored derivative of Qwen/Qwen3-VL-8B-Thinking.

Files

Hardware requirements

Requires a Blackwell-class GPU (SM ≥ 10.0) for NVFP4 inference. The INT8 ConvRot layers have broader compatibility on their own, but the file as a whole needs Blackwell because of the NVFP4 portion.

Verified loading and generation (both plain and thinking mode) via ComfyUI's CLIPLoader (type: krea2) + TextGenerate nodes on an RTX 5070 Ti.

Quantization method

Produced with convert_to_quant (ctq), NVFP4 + ConvRot INT8 tier, per a role-based layer split:

• NVFP4 (bulk, default): model.language_model.layers.N.mlp.{gate,up,down}_proj.weight for the middle 34 decoder blocks (all except the first and last).
• INT8 ConvRot (secondary): model.language_model.layers.N.self_attn.{q,k,v,o}_proj.weight for the same middle 34 decoder blocks.
• Kept at source BF16 (excluded from quantization): the entire vision tower (model.visual.* — patch embed, ViT blocks, DeepStack merger), model.language_model.embed_tokens.weight, lm_head.weight, model.language_model.norm.weight, and every tensor in decoder blocks 0 and 35 (first/last, kept high-precision for structural-position sensitivity).

Exact command:

ctq -i model_consolidated.safetensors \
    -o qwen3-vl-8b-thinking-heretic_nvfp4_convrot_int8.safetensors \
    --nvfp4 --fallback fp8 \
    --layer-config qwen3vl_8b_layer_config.json \
    --comfy_quant --save-quant-metadata

The layer-config was generated from the checkpoint's own tensor header by build_qwen3vl_layer_config.py.

Verified per-layer format distribution in the output's own _quantization_metadata: 102 nvfp4 layers, 136 int8_tensorwise (ConvRot) layers — matching the intended split exactly.

Quality has not been separately assessed beyond spot-checking generation coherence (a factual question and a math "show your reasoning" prompt in thinking mode both produced correct, coherent output) — no formal benchmark/perplexity comparison against the BF16 source has been run.

License

Apache License 2.0, inherited from the source model (Qwen/Qwen3-VL-8B-Thinking and the Kizzington/Qwen3-VL-8B-Thinking-heretic-1refusal derivative). See the Apache 2.0 license text for full terms.