Qwen3-VL-8B-Instruct SDNQ INT8 static

SDNQ static weight-only int8 quantization of Qwen/Qwen3-VL-8B-Instruct (snapshot 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b).

Built as the instruction encoder (mllm) for the Boogu Image 0.1 pipelines: the mllm/ shipped in WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static (and the Edit variant) is the stock Qwen3-VL-8B-Instruct byte for byte, so this repo is a drop-in replacement that cuts the encoder's host RAM from ~17.5 GiB (bf16) to ~9.4 GiB on disk / in RAM. It is a complete Qwen3-VL-8B-Instruct checkpoint (processor + tokenizer included) and loads with plain transformers after import sdnq.

Recipe

• SDNQ 0.2.5 static quantization, weights_dtype=int8, group_size=0 (auto), dequantize_fp32=True.
• Kept in bf16: embeddings, lm_head, multi_modal_projector, vision patch_embed and every module below SDNQ's size floor.
• Quantized layers: {"int8": 368}; quantized params: {"int8": 7517270016}; saved size 9.37 GiB (bf16 source 16.33 GiB).
• Save/load round trip verified through sdnq.loader.load_sdnq_model (all parameters bit-identical); encoder last_hidden_state vs bf16 on a mixed RU/EN/ZH prompt: {"cos_mean": 0.9716747403144836, "cos_min": 0.7031210064888, "max_abs": 30.71875, "rel_l2": 0.24023021757602692}.
• Runtime used for all numbers below: torch 2.10.0+cu130, transformers 4.57.1, diffusers 0.39.0.dev0, sdnq 0.2.5, NVIDIA GeForce RTX 4090 (RunPod, disposable pod), Python 3.12.3.

Boogu Image 0.1 Turbo A/B (same seed, bf16 encoder vs this encoder)

Pipeline BooguImageTurboPipeline from WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static @ 422ac37ad305169703fd8f847ef76418cab8363f (SDNQ uint4 transformer), 1024x1024, 4 DMD steps, text_guidance_scale=1.0, image_guidance_scale=1.0, empty_instruction_guidance_scale=0.0, dmd_conditioning_sigma=0.001, max_sequence_length=1280, sequential CPU offload, SDNQ quantized matmul on for the transformer. 12 prompts (EN/RU/ZH, typography, diagrams, long prompt); image metrics are against the bf16-encoder image of the same prompt and seed, embedding metrics compare the encoder output that is fed to the transformer. Encoder rows ending in -wo run weight-only (quantized weights dequantized per layer on the GPU, use_quantized_matmul=False); rows without the suffix also quantize the activations to int8 for the matmul, which is measurably worse for this encoder (Qwen3-VL activation outliers) and is not the recommended mode.

Host RSS is the whole pipeline process (transformer + VAE + encoder) after load with sequential offload, measured from /proc/self/status.

Per prompt (int8-wo)

Pairwise images (bf16 | quantized) are in comparison/.

Usage

import torch
import sdnq  # registers the SDNQ quantizer with transformers
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

repo = "WaveCut/Qwen3-VL-8B-Instruct-SDNQ-int8-static"
processor = AutoProcessor.from_pretrained(repo)
model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

Loader used by the aifarm draw service (weights stay on CPU, sequential offload moves layers per forward):

from sdnq.loader import load_sdnq_model
from huggingface_hub import snapshot_download

# model_cls=None: sdnq 0.2.5 crashes on an explicit transformers class (fixed in 0.2.6);
# use_quantized_matmul=False: weight-only, keeps the encoder output at cos ~0.999 vs bf16.
mllm = load_sdnq_model(snapshot_download(repo), model_cls=None,
                       dtype=torch.bfloat16, device="cpu", use_quantized_matmul=False)

Files

• model-*.safetensors, config.json (with quantization_config), quantization_config.json - SDNQ checkpoint.
• quantization_summary.json - recipe, layer counts, storage, save/load checks.
• benchmark/summary.json, benchmark/*.metrics.json - raw A/B metrics; benchmark/prompts.json - the prompt set.
• comparison/ - pairwise images and the contact sheet.
• SHA256SUMS - checksums of the checkpoint files.