ComfyUI Qwen3-VL GGUF (8B & 4B) — Unified Hybrid & Split mmproj Editions

GGUF quantizations of Qwen3-VL extracted from Comfy-Org's Qwen3-VL repackaged weights.

This repository provides two distinct formats depending on your runtime:

1. 🔥 NEW: Unified Single-File Hybrid GGUFs (*_vision_q8_0.gguf) — Combines the K-quantized LLM backbone (Q4_K_M / Q5_K_M) and the Q8_0 Vision Tower + DeepStack Mergers + 5D Conv3d shape metadata (comfy.gguf.orig_shape) inside a single file.
2. Split LLM + Standalone mmproj GGUFs — Standard two-file format for llama.cpp, llama-server, Ollama, and split VLM loaders.

---

Model Overview

• Original Weights Source: Comfy-Org/Qwen3-VL
• Underlying Architecture: Alibaba Cloud Qwen3-VL (Qwen3VLForConditionalGeneration / qwen3vl)
• Hybrid Format Vision Precision: Q8_0 Vision Tower & DeepStack Mergers + F32 LayerNorms/Biases + preserved 5D [1152, 3, 2, 16, 16] patch embedding metadata
• Split Format Projector Precision: Standalone F16 GGUF (mmproj) preserving full vision patch fidelity
• Quantization Engine: llama.cpp k-quants + custom ComfyUI-GGUF-Loader hybrid tensor packer

---

Available Files & Specifications

1. Unified Single-File Hybrid Series (Recommended for ComfyUI)

Contains both the Language Backbone (blk.*) and Vision Tower (model.visual.*) in a single GGUF file.

2. Split 8B Series (LLM + Standalone mmproj)

For llama.cpp, llama-server, or split-loader workflows.

3. Split 4B Series (LLM + Standalone mmproj)

---

Quickstart & Usage

Option A: Unified Single-File Setup

1. Download either qwen3vl_8b_q4_k_m_vision_q8_0.gguf or qwen3vl_8b_q5_k_m_vision_q8_0.gguf and place it into:

   ComfyUI/models/text_encoders/
   # or
   ComfyUI/models/clip/
   

2. Load it directly inside ComfyUI using the CLIPLoader (GGUF) node

Option B: Split Two-File Setup (llama.cpp / Split ComfyUI Loaders)

1. Place your desired language GGUF (e.g., qwen3vl-8b-Q4_K_M.gguf) into:

   ComfyUI/models/unet/
   # or
   ComfyUI/models/text_encoders/
   

2. Place the matching projector (e.g., mmproj-qwen3vl-8b-f16.gguf) into:

   ComfyUI/models/clip_vision/
   

3. Load the model using your split GGUF loader or via llama.cpp CLI/Server:

   llama-server -m qwen3vl-8b-Q4_K_M.gguf --mmproj mmproj-qwen3vl-8b-f16.gguf
   

---

Technical Verification (Hybrid Series)

Both qwen3vl_8b_q4_k_m_vision_q8_0.gguf and qwen3vl_8b_q5_k_m_vision_q8_0.gguf have been validated end-to-end:

• 750 Total Tensors: 399 LLM tensors (Q4_K_M or Q5_K_M) + 351 Vision tensors (Q8_0 weights + F32 norms/biases).
• 5D Patch Embedding Metadata: Includes comfy.gguf.orig_shape.model.visual.patch_embed.proj.weight = [1152, 3, 2, 16, 16] for 3D temporal-spatial patch projection.
• DeepStack Mergers: All 3 DeepStack spatial mergers ([4096, 4608]) and final vision-to-LLM merger verified with zero NaN/Inf values during multimodal forward passes.

---

Acknowledgements

• Alibaba Cloud Qwen Team for the foundational Qwen3-VL models.
• Comfy-Org for packaging the unified BF16 safetensors weights.
• Georgi Gerganov and the GGML/llama.cpp contributors for GGUF architecture and quantization tooling.