Disty0/Z-Image-SDNQ-4bit-dynamic-hadamard-svd-r32

Dynamic 4 bit quantization of Tongyi-MAI/Z-Image using SDNQ.

This model uses per layer fine grained quantization.
What dtype to use for a layer is selected dynamically by trial and error until the std normalized mse loss is lower than the selected threshold.

Minimum allowed dtype is set to uint4 and std normalized mse loss threshold is set to 8e-3.
This created a mixed precision model with uint4 and int5 dtypes.
Hadamard rotations are enabled with Hadamard Group size 128.
SVD quantization is enabled with SVD rank 32.

Usage:

pip install sdnq>=0.1.9

Note: Minimum SDNQ version required for this model is v0.1.9! Otherwise you will get garbled output.

import torch
import diffusers
from sdnq import SDNQConfig # import sdnq to register it into diffusers and transformers

pipe = diffusers.ZImagePipeline.from_pretrained("Disty0/Z-Image-SDNQ-4bit-dynamic-hadamard-svd-r32", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

# Generate image
prompt = "Young Chinese woman in red Hanfu, intricate embroidery. Impeccable makeup, red floral forehead pattern. Elaborate high bun, golden phoenix headdress, red flowers, beads. Holds round folding fan with lady, trees, bird. Neon lightning-bolt lamp (⚡️), bright yellow glow, above extended left palm. Soft-lit outdoor night background, silhouetted tiered pagoda (西安大雁塔), blurred colorful distant lights."
negative_prompt = "" # Optional, but would be powerful when you want to remove some unwanted content

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=1024,
    width=1024,
    cfg_normalization=False,
    num_inference_steps=20,
    guidance_scale=4,
    generator=torch.manual_seed(42),
    callback_on_step_end=callback_func,
).images[0]
image.save("z-image-turbo-sdnq-4bit-dynamic-hadamard-svd-r32.png")

Original BF16 vs SDNQ quantization comparison: