🐷pig architecture gguf clip

• natively trained multipurpose clip for gk engine
• low vram alternative/substitute of t5xxl, umt5xxl, etc.
• 80-90% smaller, more efficient and cost effective (hopefully)

rationale

since LLM in diffusion model basically serves as text encoder, which means its primary goal/function is delivering a prompt to the generator precisely, no talk, no deep reasoning, no long response, make it as simple as possible

limitation/cutoff

we stop the training when similarity reaches .95 or over 20k cycles and similarity no less than .75

how it works

simply replace --t5xxl t5xxl.gguf with --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf

examples

test it with pixart

gguf --diffusion-model pixart-nvfp4.gguf --vae pig_pixart_vae_fp16-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --diffusion-fa -v -o out.png

test it with sd-lite

gguf --diffusion-model model.gguf --vae vae.gguf --clip_l clip_l.gguf --clip_g clip_g.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --steps 8 --cfg-scale 1 --sampling-method euler --clip-on-cpu --diffusion-fa -v -o out.png

umt5 adapter

test it with wan2.1 or wan2.2

gguf -M vid_gen --diffusion-model wan2.1_t2v_1.3b-q4_0.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_umt5_adapter-f16.gguf -p "a pig moving quickly in a beautiful winter scenery nature trees sunset tracking camera" --cfg-scale 6.0 --sampling-method euler -v -n "blurry ugly bad" -W 480 -H 480 --diffusion-fa --offload-to-cpu --video-frames 14 -o out.avi

qwen3vl-layerwise adapter

test it with krea2

gguf --diffusion-model krea2_turbo-q2_k.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_layerwise_adapter-f16.gguf -p "a lovely pig holding a sign says Hi" --cfg-scale 1.00 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3vl-fibo2 adapter

test it with fibo2

gguf --diffusion-model fibo2-edit-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_fibo2_adapter-f16.gguf -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

gguf --diffusion-model fibo2-edit-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_fibo2_adapter-f16.gguf --llm_vision mmproj-qwen3vl-4b-it-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --seed 1 --sampling-method euler --diffusion-fa -v -o out.png

qwen3-4b adapter

test it with x-image

gguf --diffusion-model  x_image-nvfp4.gguf --vae pig_flux_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3_4b_adapter-f16.gguf -p "cute anime style girl with pinky messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron, it is a postcard held by a hand in front of a beautiful realistic city at sunset and there is cursive writing that says PIG" --cfg-scale 1.0 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3vl-4b adapter

test it with mageflow

gguf --diffusion-model mageflow-edit-turbo-nvfp4.gguf --vae pig_mageflow_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_4b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-4b-it-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

qwen3vl-4b-it adapter

test it with flux3

gguf --diffusion-model flux3-image-klein-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_4b_it_adapter-f16.gguf -p "a sheep in sunglasses" --cfg-scale 1 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

image edit (no vision clip needed)

gguf --diffusion-model flux3-image-klein-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_4b_it_adapter-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

qwen3vl-8b adapter

test it with qwen-image2.1

gguf --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf -p "an anime pig holding a sign says GGUF" -W 512 -H 512 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

gguf --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-8b-it-q8_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

qwen3.5-9b adapter

test it with moss

gguf --diffusion-model moss-image-sft-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3.5_9b_adapter-q8_0.gguf -W 512 -H 512 -p "a sheep in sunglasses" --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

gguf --diffusion-model moss-image-sft-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3.5_9b_adapter-q8_0.gguf --llm_vision mmproj-qwen3.5-9b-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --sampling-method euler --diffusion-fa -v -o out.png

llada adapter

test it with llada

gguf --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q8_0.gguf --llm-adapter pig_llada_adapter-q8_0.gguf -p "a pig in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

gguf --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q4_k_m.gguf --llm-adapter pig_llada_adapter-f16.gguf --llm_vision pig_llada_sigvq-q4_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

ming adapter

test it with ming

gguf --diffusion-model ming-image-0.1-design-nvfp4.gguf --vae pig_ming_image_vae_bf16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_ming_adapter-f16.gguf -p "a pig in sunglasses" -W 512 -H 512 --steps 12 --diffusion-fa -v -o out.png

edit image with vision clip (optional)

gguf --diffusion-model ming-image-0.1-design-nvfp4.gguf --vae pig_ming_image_vae_bf16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_ming_adapter-f16.gguf --llm_vision pig_ming_vision-nvfp4.gguf --ref-image sheep.png -p "a sheep in sunglasses" -W 512 -H 512 --steps 12 --diffusion-fa -v -o out.png

layer decompose with vision clip (optional)

gguf --diffusion-model ming-image-0.1-design-layer-nvfp4.gguf --vae pig_ming_image_vae_bf16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_ming_adapter-f16.gguf --llm_vision pig_ming_vision-q8_0.gguf --ref-image card_making_input.png --prompt-file card_making_prompt.txt -W 512 -H 512 --steps 12 --diffusion-fa -v -o out_ming_layer.png

both ggk (pip install ggk) and gguf (pip install gguf-core) commands work; ggk serves as a nightly version of gguf

Reference

pig engine - the new gguf compute kernels (gk)

gguf trainer - how to train your own adapter