中文Sign up

RachidAR/gemma-4-31B-it-qat-Q4_0-Q4emb-MTP-assistant-gguf

Source: 🤗mit31B activated1.6 GBGGUF✓ 3 checksumsupdated 134d ago
or submit a torrent

Gemma 4 31B QAT MTP Assistant GGUF

GGUF quantizations converted from google/gemma-4-31B-it-qat-q4_0-unquantized-assistant.

Tested with llama.cpp b9549+ (Gemma 4 MTP support).

Recommendations

  • Q4_0-Q4emb — recommended for most users
  • Q4_0-Q6emb — standard llama.cpp quantization

Files

  • gemma-4-31b-qat-it-assistant-bf16.gguf
  • gemma-4-31b-qat-it-assistant-Q4_0-Q4emb.gguf
  • gemma-4-31b-qat-it-assistant-Q4_0-Q6emb.gguf

Notes

Q4_0-Q4emb keeps token_embd.weight in Q4_0, making it closer to the original QAT layout.

This follows a similar approach used by recent Unsloth Gemma QAT releases. Initial testing showed comparable draft acceptance rates while reducing model size.

Example

llama-server \
  -m gemma-4-31B-it-qat-Q4_0.gguf \
  -md gemma-4-31b-qat-it-assistant-Q4_0-Q4emb.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

Recommended values:

  • --spec-draft-n-max 2 for general use
  • --spec-draft-n-max 3 for coding workloads
  • --spec-draft-n-max 4 may help on highly repetitive tasks