DEPRECIATED!

This model was surpassed by:

https://huggingface.co/magiccodingman/Granite-4.0-H-350M-Unsloth-MagicQuant-Hybrid-GGUF

Use the new version. This shown MXFP4 hybrid is no longer viable in comparison, nor really useable. The data collected is good for understanding and research, but it's not as good for real use.

Unsloth - Granite 4.0 H 350M MXFP4 Hybrid GGUF

Dense model utilizing MXFP4_MOE with hybrid weights on a dense model. Achieving interesting results that show smaller file size, more TPS, and near lossless precision.

Use The Following Models!

• MXFP4_MOE-output_q6_k-router_gate_emb_f16 (This is the special version)

  29.7% smaller than F16 • 1652.1 TPS • 0.04959% precision loss compared to F16

Why is this version special? Because precision loss on tiny models like this affect the model to the extreme. To achieve ~30% smaller size at this small of a precision loss is the exact trade off desired when needing to minimize the models size. This is the primary variant suggested for this model.

---

Stats compared against the standard Q8_0 (precision loss still compared to F16)

• MXFP4_MOE-output_q6_K-router_gate_emb_q6_K

  9.1% smaller than Q8 • 1721.67 TPS • 0.214% precision loss

---

This repository contains a set of hybrid MXFP4 quantized GGUF models designed to explore a surprising discovery:

A carefully targeted combination of MXFP4 + high-precision embeddings/output weights can deliver near-Q8 accuracy with Q4–Q6 level throughput and smaller file sizes than Q8.

Unlike pure MXFP4, which heavily degrades dense models. This hybrid method selectively protects tensors that matter most for semantic stability, while allowing MXFP4 to accelerate everything else.

This is experimental. And should be treated as such. I am more than encouraging people to use this model and leave feedback! Though precision loss seemed near lossless, did the hybrid models act strange in certain situations? Worse or better on some topics compared to the original model? Did it do better/worse overall on everything? I'd love to hear back from others!

---

The Magic Model

MXFP4_MOE-output_q6_k-router_gate_emb_f16

(29.7% smaller than F16 • 1652.1 TPS • 0.04959% precision loss compared to F16)

This... this is hot.. if I do say so myself.

The following was the conversion script:

llama-quantize \
  --tensor-type token_embd.weight=F16 \
  --tensor-type output.weight=Q6_K \
  --tensor-type 'router.*'=F16 \
  --tensor-type 'gate.*'=F16 \
  "Path_To_F16_GGUF.gguf" \
  "Path_To_GGUF.gguf" \
  mxfp4_moe

MXFP4_MOE-output_q6_K-router_gate_emb_q6_K

(9.1% smaller than Q8 • 1721.67 TPS • 0.214% precision loss )

This version created beat out everything in every way in the MXFP4 hybrid family created. Out of the batch, this MXFP4 hybrid was the only worth considering to utilize.

The following was the conversion script:

llama-quantize \
  --tensor-type token_embd.weight=Q6_K \
  --tensor-type output.weight=Q6_K \
  --tensor-type 'router.*'=Q6_K \
  --tensor-type 'gate.*'=Q6_K \
  "Path_To_F16_GGUF.gguf" \
  "Path_To_GGUF.gguf" \
  mxfp4_moe

---

MXFP4_MOE Hybrid Naming Scheme & Synopsis

Multiple different combinations of converted models were created. The results were interesting to say the least. The following table will explain my naming scheme to what was done to the model to create it.

The results achieved were interesting to say the least. It was a brute force game of mass creating models with hybrid methods to find combinations that didn't cause too much noise and paired well with MXFP4.

This repo showcases the converted models, whether good or bad that was created. But, I have been testing other models in different combinations as well. The winning hybrid combinations shown in this repo DOES NOT always equate to the same results on different models.

Some models do better or worse with different kinds of combinations. It depends if it's dense, MOE, and much more. Many times the results surprise me. Many models no matter the combination will not play nice with MXFP4. At least with the methods shown here.

---

Benchmark Methodology

All models were tested with a unified automated harness using llama.cpp tools.

Included tests:

• Throughput:
  llama-bench with descending GPU offload (-ngl 35 → 0) and automatic OOM retry.
  Highest successful TPS is recorded.

• Perplexity:
  Three domains: general, code, math.
  Each uses an auto-generated corpus of ~32k tokens.
  Perplexity is computed with llama-perplexity at 2048-token context.
  Same GPU retry logic as above.

• Precision loss:
  Each model is compared to its family F16 baseline.
  Precision-loss % is computed for all PPL domains, plus an averaged score.
  Models are ranked by this metric.

---

Table - Overview of Results

Comparing to F16.

• All percentages compared against the selected family F16 baseline.

---

Table - File Size + TPS + Avg Precision Loss

• Bench NGL was 35
• Utilized CUDA

---

Table - PPL Columns

• gen = ppl_general
• gen_er = ppl_general_error
• code = ppl_code
• code_er = ppl_code_error
• math = ppl_math
• math_er = ppl_math_error

---

Table - Precision Loss Columns

• loss_general = precision_loss_general_pct
• loss_code = precision_loss_code_pct
• loss_math = precision_loss_math_pct