Qwen3.5-122B-A10B-abliterix UD-Q4_K_XL GGUF

Overview

Qwen3.5-122B-A10B-abliterix-UD-Q4 K XL-GGUF is a GGUF release for llama.cpp-compatible runtimes and local inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.gguf (2 files)
• Additional configuration, tokenizer, processor, or shard files (2 visible artifacts total)

Quick start

llama.cpp

Download a .gguf file that fits your available memory, then run it with a current llama.cpp build:

llama-cli \
  -m /path/to/model.gguf \
  -p "Write a concise technical summary."

For vision or any-to-any models, download the matching multimodal projection file when one is provided and follow the source model's modality-specific instructions.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

User-Defined (UD) mixed-precision quantization of Qwen3.5-122B-A10B-abliterix for llama.cpp.

What is UD (User-Defined) Quantization?

UD quantization is a per-tensor mixed-precision approach that goes beyond blanket quantization methods (like Q4_K_M or Q8_0 applied uniformly). Instead of using the same quantization type for every tensor, UD assigns an individually chosen type to each tensor based on its role in the model architecture and its sensitivity to precision loss.

How it differs from standard quantization

The UD methodology for this model

1. Architecture analysis: Qwen3.5-122B-A10B is a Mixture-of-Experts (MoE) model with 48 layers, 256 experts per layer (8 active), shared expert FFN, and SSM (state-space model) components. Each tensor was classified by its function:

   • Attention weights (attn_qkv, attn_gate, attn_q, attn_k, attn_v, attn_output): Quantized to Q8_0 — attention projections are highly sensitive to precision loss and benefit from near-lossless compression.

   • Expert FFN up/gate weights (ffn_up_exps, ffn_gate_exps): Quantized to Q4_K — these are the largest tensors in the model (256 experts x 3072x1024 each) and dominate the model size. Q4_K provides an excellent size/quality tradeoff for activation functions.

   • Expert FFN down weights (ffn_down_exps): Quantized to Q5_K — the down-projection is slightly more sensitive than up/gate, so Q5_K preserves more information here.

   • Shared expert FFN (ffn_down_shexp, ffn_up_shexp, ffn_gate_shexp, ffn_gate_inp_shexp): Quantized to Q8_0 — the shared expert processes every token, making it critical for quality.

   • Router/gating (ffn_gate_inp, ffn_gate_inp_shexp): Kept at F32 — the router decides which experts to activate. Even small precision errors here can cause misrouting, severely degrading output quality.

   • SSM components (ssm_a, ssm_conv1d, ssm_dt.bias, ssm_norm): Kept at F32 or Q8_0 — these are small tensors that play a structural role in the hybrid attention/SSM layers.

   • Norms (attn_norm, post_attention_norm, output_norm): Kept at F32 — normalization layers are tiny but critical for numerical stability.

   • Token embeddings (token_embd.weight): Quantized to Q8_0.

   • Output projection (output.weight): Quantized to Q8_0.

2. Importance matrix: A partial importance matrix (imatrix) was computed on coding-focused calibration data using llama-imatrix. This guides the k-quant block selection within each tensor, prioritizing precision on the most activation-sensitive dimensions. The imatrix was computed at n_ctx=512 across multiple calibration chunks before the quantization run.

3. Quantization: llama-quantize from llama.cpp was used with --tensor-type-file to apply the per-tensor type map, combined with --imatrix for importance-weighted block selection within each quantized tensor.

Resulting size

Usage with llama.cpp

llama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \
  -ngl 999 \
  -c 32768 \
  -fa on \
  --reasoning auto

Multi-GPU tensor split example (3x RTX 3090 + 2x RTX 3060)

llama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \
  -ngl 999 \
  -ts 24,24,12,12,24 \
  -c 32768 \
  -ctk q8_0 -ctv q8_0 \
  -fa on \
  --reasoning auto

Provided by

• Quantized by: groxaxo
• Base model: wangzhang/Qwen3.5-122B-A10B-abliterix
• Quantization engine: llama.cpp build 8736