Qwen-Image-2.1-Turbo (Nunchaku SVDQuant NVFP4 Rank-128 with Channel-Wise Macro Scaling) 🌟🚀

This repository provides the high-fidelity SVDQuant NVFP4 (Rank 128, 32 Alternating SVD Iterations) quantized transformer for Alibaba's Qwen-Image-2.1-Turbo. Forged with channel-wise Level-0 macro scaling (wcscales) and optimized low-rank error absorption, it enables ultra-fast 8-step flow-matching generation and in-context image editing with Nunchaku on NVIDIA Blackwell (sm_120), Ada Lovelace, and Hopper architectures.

---

⚡ Highlights

• 100% Resident VRAM Execution on Consumer 16GB GPUs:
  • Weights compressed from 30.2 GB (BF16) down to 4.54 GB (NVFP4) — a 6.65x reduction in transformer memory footprint!
  • Runs entirely resident in VRAM on consumer GPUs (e.g. NVIDIA GeForce RTX 5060 Ti 16GB, RTX 4080 16GB, RTX 4090 24GB) with over 11 GB free VRAM headroom on a 16GB card.
• Rank-128 SVD Quantization Conquers Complex Facial Distortion:
  • Quadrupled the SVD low-rank capacity from standard Rank 32 to Rank 128, trained over 32 alternating SVD error compensation iterations.
  • Slashed relative $L_2$ projection reconstruction residual by -44.7% (from $0.1305$ down to $0.07219$) and elevated Cosine Similarity to $0.99740$ across all 224 NVFP4 linear layers.
  • Completely eliminates the smeared eyes, smudged foreheads, and distorted facial geometry that lower-rank (Rank 32) models encounter during complex multi-subject in-context photo edits (e.g., adding multiple children in front of adults in high-resolution family photos).
• Blazing Fast 8-Step Flow-Matching Latency:
  • 8-step Text-to-Image (1024x1024) in ~12–14 seconds (1.65 it/s) on a single RTX 5060 Ti — over 3x faster than 25-step base models.
  • Multi-subject in-context image editing in 12.08 seconds (1024x576, 8 steps).
• Channel-Wise Level-0 Macro Scaling (wcscales):
  • Incorporates 224 channel-wise scale vectors across all transformer blocks, protecting outlier activation channels and eliminating color bleed and subscale saturation.
• Procedural Geometry Excellence:
  • Benchmarked on solid geometric primitives (e.g. red square on solid cyan studio background): achieves a background Laplacian variance of $12.56$ (target $< 15.0$) with zero cobblestone or tiling artifacts.
• Diffusers PR #14950 Flow-Matching Compatibility:
  • Fully integrated with Diffusers flow-matching schedulers, automatically handling sample_sigmas for 8-step distilled generation.
• Designed for Pairing with Heretic Text Encoder:
  • Perfectly complements Qwen21_Text_Encoder_Heretic to eliminate false refusals, vector deflection, and unnatural anatomical postures.

---

📊 Empirical Benchmarks & Hardware Profiling (RTX 5060 Ti 16GB)

1. Latency & Resource Utilization

---

2. SVD Rank Capacity vs. Matrix Reconstruction Fidelity

Evaluated across all 224 linear layers in the 32 transformer blocks (tested on Block 0 img_mlp.proj $12288 \times 4096$ and attn.to_q $4096 \times 4096$):

---

3. Blackwell SM120 Hardware W4A4 GEMM Profiling

Direct CUDA hardware latency measurement of Nunchaku's svdq_gemm_w4a4_cuda kernel on RTX 5060 Ti:

[!NOTE] Moving from Rank 32 to Rank 128 adds only ~29 milliseconds to the entire 8-step generation while cutting reconstruction error by 44.7%!

---

🚀 Quickstart Usage

1. Requirements & Nunchaku / Diffusers Setup

Official PyPI releases of Nunchaku (v0.16.x) do not yet include the NunchakuQwenImage21Transformer2DModel class, and older Diffusers releases may lack QwenImage21Pipeline flow-matching sample_sigmas support.

We provide multiple zero-friction options in the extras/packages/ directory:

• Option A (Drop-in Class — Zero Reinstall): If you already have Nunchaku installed, simply import directly from this repo:

  from extras.transformer_qwenimage21 import NunchakuQwenImage21Transformer2DModel
  

• Option B (Prebuilt Wheels):

  # Diffusers with Qwen-Image-2.1 support and VAE decode memory offload
  pip install extras/packages/diffusers-0.41.0.dev0-py3-none-any.whl
  
  # Patched Nunchaku wheel for Linux x86_64 / Python 3.12 (CUDA 12/13, sm_120/sm_110)
  pip install extras/packages/nunchaku-1.2.1+cu13.0torch2.12-cp312-cp312-linux_x86_64.whl
  

• Option C (Git Bundles / Unified Diffs): Full Git bundles (extras/packages/nunchaku-qwenimage21.bundle, extras/packages/diffusers-qwenimage21.bundle) and unified .diff patches are also included for inspecting or building from source. See extras/packages/README.md for complete details.

2. Loading with Diffusers & Nunchaku

import torch
from diffusers import QwenImage21Pipeline
# If using the drop-in class from extras:
from extras.transformer_qwenimage21 import NunchakuQwenImage21Transformer2DModel
# Or if using the packaged nunchaku wheel/bundle:
# from nunchaku.models.transformers.transformer_qwenimage21 import NunchakuQwenImage21Transformer2DModel
from PIL import Image

device = "cuda:0"

# 1. Load forged Rank-128 NVFP4 DiT directly into resident VRAM
transformer = NunchakuQwenImage21Transformer2DModel.from_pretrained(
    "svdq-fp4_r128-qwen-image-2.1.safetensors",
    device=device,
    torch_dtype=torch.bfloat16,
)

# 2. Assemble complete Turbo pipeline
pipeline = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1-Turbo",
    transformer=transformer,
    torch_dtype=torch.bfloat16,
)
pipeline.vae = pipeline.vae.to(device)
pipeline.vae.enable_tiling()

# 3. Fast 8-step generation (1024x1024 in ~13 seconds!)
image = pipeline(
    prompt="A magnificent snow leopard resting on a Himalayan rocky ledge, golden hour sunlight, 8k resolution photo",
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=1.0,
).images[0]

image.save("snow_leopard.png")

---

3. Multi-Subject In-Context Image Editing

# Load base image to edit
init_image = Image.open("family_photo.png").convert("RGB")

edited_image = pipeline(
    prompt="Add 3 little girls standing in front of adults in this photo, smiling, wearing warm jackets",
    image=init_image,
    width=1024,
    height=576,
    num_inference_steps=8,
    guidance_scale=1.0,
).images[0]

edited_image.save("family_photo_edited.png")

---

4. Optional: Pairing with an Uncensored Text Encoder for Enhanced Prompt Following 🛡️✨

Multi-modal diffusion models like Qwen-Image-2.1 rely heavily on the visual-language text encoder (Qwen3-VL-8B) for conditional guidance. However, standard stock text encoders often harbor internal refusal and suppression vector projections that inadvertently trigger on innocent, natural human prompts (e.g., family beach vacations, swimwear, children playing outdoors, medical contexts, or dynamic athletic stances).

When these guardrails activate, they deflect the text embedding trajectory into "hesitation" manifolds—causing:

1. Compounded Anatomical Glitches: Extra limbs, distorted fingers, or twisted postures.
2. Facial & Subject Omission: Ignoring key subject counts (e.g. failing to render children when requested alongside adults).
3. Geometric Searing / Inpainting Artifacts: Blurry boundary seamlines during in-context edits.

Solution: Uncensored / Abliterated Text Encoder (Qwen21_Text_Encoder_Heretic)

Pairing this Rank-128 NVFP4 transformer with an uncensored text encoder (such as Qwen21_Text_Encoder_Heretic) eliminates refusal vector deflection and delivers pinpoint prompt following:

• Natural Scene Adherence: Correct two-legged anatomy, clean athletic stances, and natural clothing fitting the scene context.
• Complex Multi-Subject Rendering: Accurately renders exact counts of subjects (e.g., "Add 3 little girls standing in front of adults in this photo") with smiling, photorealistic facial geometry.

Loading with an Uncensored Text Encoder in Python:

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from diffusers import QwenImage21Pipeline

# Load custom/uncensored text encoder
text_encoder_path = "path/to/Qwen21_Text_Encoder_Heretic/text_encoder"
text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
    text_encoder_path,
    torch_dtype=torch.bfloat16,
    device_map="cpu",  # or stream with extras/stream_encoder.py
)

pipeline = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1-Turbo",
    transformer=transformer,
    text_encoder=text_encoder,
    torch_dtype=torch.bfloat16,
)

Launching the Packaged Server with Uncensored Conditioning:

# Launch high-throughput server on port 4503 with the uncensored text encoder:
./extras/imagegen_qwen21_turbo_nvfp4.sh 4503 /path/to/Qwen21_Text_Encoder_Heretic

---

📦 Packaged Tooling (extras/)

All necessary inference, streaming, quantization, and diagnostic tools are packaged directly in the extras/ directory of this repository:

🚀 Production Inference & Serving

• extras/packages/: Prebuilt wheels (.whl), standalone Git bundles (.bundle), and unified diff patches (.diff) for both Nunchaku and Diffusers.
• extras/transformer_qwenimage21.py: Standalone drop-in class for NunchakuQwenImage21Transformer2DModel compatible with any existing Nunchaku installation.
• extras/imagegen_qwen21_turbo_nvfp4.sh: One-click production server launcher for resident VRAM serving on port 4503 (compatible with ThinkMAX or standalone).
• extras/ImageEditServer.py: High-concurrency FastAPI/Uvicorn server supporting OpenAI-compatible Text-to-Image (/v1/images/generations) and Multimodal In-Context Image Editing (/v1/images/edits).
• extras/QwenImage21NVFP4Backend.py: Production backend managing 100% resident NVFP4 DiT memory layout, VAE tiling, and optional PCIe layerwise text encoder streaming.
• extras/QwenImage21Backend.py: Reference unquantized BF16 backend for baseline quality, speed, and numerical divergence comparisons.
• extras/stream_encoder.py: Zero-quality-loss layerwise weight streaming engine for Qwen3-VL-8B allowing full text conditioning on VRAM-constrained systems.

⚡ Quantization & Packing Engine

• extras/stream_quantize_qwen21.py: The core streaming quantizer implementing randomized torch.svd_lowrank alternating error compensation (32 iterations) and channel-wise Level-0 macro scaling (wcscales).
• extras/packer.py: Nunchaku NVFP4 parameter packer and state dict converter.
• extras/quantize_turbo.sh: Full automated shell script to quantize all 32 transformer blocks and merge shards into a consolidated checkpoint.

🧪 Validation & Benchmark Suites

• extras/test_turbo_r128_edit.py: Multi-subject in-context photo edit test script with face sharpness evaluation across multiple seeds.
• extras/test_turbo_geometry.py: Procedural geometry evaluator testing background smoothness (Laplacian variance), edge acuity, and color variance on vector primitives.

---

📜 Citation & Acknowledgments

• Qwen-Image-2.1-Turbo: Qwen Team, Alibaba Cloud.
• SVDQuant & Nunchaku: MIT HAN Lab (SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models).
• Quantization & Rank-128 Alternating SVD Optimization: Oleg K. / Nikola Seeker Project.