Qwen3.6-35B-A3B-MXFP4-MOE-GGUF

MXFP4_MOE GGUF quantization of Qwen3.6-35B-A3B, Alibaba Cloud's efficient 35B MoE multimodal foundation model (3B activated) with 262K context, 256 experts, and Gated DeltaNet hybrid attention.

About MXFP4_MOE

What is MXFP4_MOE?

MXFP4 (Microscaling FP4) is a 4-bit floating-point quantization format standardized by the Open Compute Project (OCP) in 2024, backed by AMD, NVIDIA, Microsoft, Meta, and OpenAI. It uses an E2M1 layout (1 sign, 2 exponent, 1 mantissa bit) with block-wise shared 8-bit scaling factors (block size 32), preserving dynamic range far beyond integer 4-bit formats.

In llama.cpp, MXFP4_MOE applies MXFP4 quantization specifically to MoE expert weights, while keeping router and shared expert layers in Q8_0 for precision where it matters most.

Why use MXFP4_MOE?

1. MoE-optimized: Expert weights quantized to MXFP4, router/shared layers in Q8_0
2. Cross-platform: Supported on AMD MI350/MI355, NVIDIA Blackwell, and OCP-compliant hardware
3. Memory efficiency: ~4.57 BPW, 35B MoE fits in ~18 GB
4. Open standard: OCP Microscaling Formats v1.0 specification, vendor-neutral

Files

Quantization Details

Model Description

Qwen3.6-35B-A3B features:

• MoE architecture: 256 experts with 8 experts per token, 35B total / 3B activated
• Hybrid attention: Alternating Gated DeltaNet (linear attention) and full attention layers
• 262K context window: Native support for 262,144 token sequences
• Vision capabilities: Built-in 27-layer ViT for image understanding
• Multi-Token Prediction: Trained with MTP for speculative decoding
• Multilingual: Strong performance across 201 languages

Usage

llama.cpp (CLI)

# Text + Image
llama-cli   -m qwen3.6-35b-a3b-mxfp4_moe.gguf   --mmproj qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf   --image photo.jpg   -p "Describe this image in detail"

# Text only
llama-cli   -m qwen3.6-35b-a3b-mxfp4_moe.gguf   -p "Explain quantum computing in simple terms"   -n 512

# OpenAI-compatible server
llama-server   -m qwen3.6-35b-a3b-mxfp4_moe.gguf   --mmproj qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf   --port 8080

llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF",
    filename="qwen3.6-35b-a3b-mxfp4_moe.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion([
    {"role": "user", "content": "What is the capital of France?"}
])
print(response["choices"][0]["message"]["content"])

Direct download

from huggingface_hub import hf_hub_download

for filename in ["qwen3.6-35b-a3b-mxfp4_moe.gguf", "qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf"]:
    hf_hub_download(
        repo_id="FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF",
        filename=filename,
        local_dir="./models"
    )

Quantization Pipeline

1. Download source weights
   huggingface_hub.snapshot_download("Qwen/Qwen3.6-35B-A3B")

2. Convert text model to F16 GGUF
   convert_hf_to_gguf.py --outtype f16

3. Extract vision encoder
   convert_hf_to_gguf.py --mmproj --outtype f16

4. Quantize to MXFP4_MOE
   llama-quantize qwen3.6-35b-a3b-f16.gguf qwen3.6-35b-a3b-mxfp4_moe.gguf MXFP4_MOE

Hardware

License

The original Qwen3.6-35B-A3B is released under Apache 2.0. These quantized weights inherit that license.