Huihui4-48B-A4B-abliterated GGUF

This repository contains GGUF quantizations of huihui-ai/Huihui4-48B-A4B-abliterated, plus the matching multimodal projector for llama.cpp-based inference.

The original model is a Gemma 4 multimodal MoE with 256 experts and 8 active experts per token. According to the upstream release, experts 1-128 come from huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated and experts 129-256 come from TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill.

These files were exported from the original safetensors model with the Unsloth GGUF export pipeline and validated with llama.cpp.

What Is In This Repo

Benchmark Setup

All benchmarks below were run with:

• llama.cpp build d132f22fc (8739)
• RTX 4090 24 GB + RTX 3090 24 GB
• AMD Ryzen 7 9800X3D
• flash attention enabled
• split mode layer
• q8_0 KV cache

Speed was measured with llama-bench at 4096 prompt tokens and 256 generated tokens. Perplexity was measured with llama-perplexity on raw Wikitext-2 validation text at n_ctx=4096.

The perplexity values are only meant as relative comparisons between these quants under one fixed setup. This is an instruction-tuned multimodal chat model evaluated on raw Wikitext-2 text, so the absolute values should not be treated as a general LM leaderboard score.

Benchmark Results

Recommended Picks

• Use Q4_K_M if you want the default recommendation.
• Use Q6_K if you want the best quality that still fits cleanly on a strong dual-24 GB setup.
• Use Q8_0 only if you are comfortable with partial CPU offload or much larger available memory.
• Use Q3_K_M or Q2_K only when memory is the priority and you accept a major quality hit.

llama.cpp Usage

Multimodal inference requires both the main quant and the projector file. On the tested runtime, Gemma 4 chat formatting also needed --jinja. If you need OpenAI-compatible tool calling from llama-server, do not pass --skip-chat-parsing.

Example llama-server launch:

llama-server \
  -m Huihui4-48B-A4B-abliterated.Q4_K_M.gguf \
  --mmproj Huihui4-48B-A4B-abliterated.BF16-mmproj.gguf \
  --jinja \
  --reasoning off \
  -fa on \
  -sm layer \
  -dev CUDA0/CUDA1 \
  -ngl 99 \
  -c 32768 \
  -np 1 \
  --host 127.0.0.1 \
  --port 8080

Example text-only llama-cli launch:

llama-cli \
  -m Huihui4-48B-A4B-abliterated.Q4_K_M.gguf \
  --jinja \
  -fa on \
  -sm layer \
  -dev CUDA0/CUDA1 \
  -ngl 99 \
  -c 32768

Notes

• This is a quantized GGUF release of the original model, not the earlier REAP-pruned experiment.
• For image input, the BF16-mmproj.gguf file is required regardless of which quant you choose.
• The Q8 benchmark above is intentionally labeled as partially CPU offloaded because it did not fit as a clean all-GPU run on the tested 4090 + 3090 machine.

Credits

• Original model: huihui-ai/Huihui4-48B-A4B-abliterated
• Expert sources: huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated and TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill
• GGUF export pipeline: Unsloth
• Runtime and benchmarks: llama.cpp