QwenPaw Flash 9B Heretic MTP — GPTQ-Pro INT4 g64

This repository contains a quality-oriented symmetric INT4 group-64 quantization of SC117/QwenPaw-Flash-9B-heretic-MTP-GGUF.

Exact source provenance

• Source file: QwenPaw-Flash-9B-heretic-MTP-F16.gguf
• Source revision: 805a37e8d1277f7db31347283017afb26a04e5a4
• Source SHA-256: 28039894f5e2388ef9a12f98bd33a9655014f9766f28f5bfe6677ca4e803a77f
• Source size: 18,407,320,992 bytes
• Output file: QwenPaw-Flash-9B-heretic-MTP-GPTQPro-INT4-g64.gguf
• Output SHA-256: 7b2625a843b80186586e1bd895364b83acd10c755b4e7267b9e0ad18cb59fb16
• Output size: 8,471,015,072 bytes (7.889 GiB)

Important: despite the upstream filename containing F16, inspection found 258 BF16 tensors and 184 F32 tensors. No substitute model weights were used.

What was improved

This was a matched A/B quantization experiment, not a comparison of different bit widths.

Both candidates used:

• symmetric INT4
• group size 64
• sequential group indices
• damping 0.05
• identical source weights
• identical calibration and held-out data
• identical packed decoder byte budget
• strict numerical checks
• zero emergency CPU Hessian fallbacks
• no GPTAQ, FOEM, EoRA, rotation, or mixed-precision promotion

The quality candidate changed only three recipe switches as a package:

1. GAR / group-aware activation ordering: off -> on
2. MSE clipping: 0 -> exponent 2
3. activation-weighted MSE: off -> on

Full-model quality result

All 32 decoder layers and all 200 selected decoder linears were quantized. The held-out evaluation scored 7,168 next-token positions across 32 conversations.

Main finding: source-to-quant KL divergence decreased by 30.41% at the same INT4 group size and the same packed decoder byte budget.

The paired conversation-bootstrap baseline-minus-quality KL improvement was 0.01449751 nats/token, with a 95% interval of [0.00588705, 0.02662250]. The interval is positive in this experiment.

The NLL improvement was much smaller: 0.00080771 nats/token, 95% interval [-0.01161773, 0.01413270]. Because this interval includes zero, predictive-loss improvement is not statistically established here.

Perplexity changed by 0.081% and source top-1 agreement increased by 0.419 percentage points.

Per-domain result

All three domains reduced source KL. Chat and tool NLL moved slightly in the wrong direction, so this is not claimed as a blanket task-accuracy improvement.

Calibration and evaluation design

• Calibration: 32 conversations, 16,272 tokens
• Held-out: 32 conversations, 8,192 input tokens
• Scored next-token positions: 7,168
• Mix: 50% code, 25% chat, 25% tools by row count
• Calibration and held-out conversation hashes were verified disjoint
• Exact GGUF chat template and tokenizer were frozen before the comparison
• One calibration seed was used

This is a source-fidelity experiment, not a HumanEval score or production tool-calling benchmark.

MTP was preserved and executed

The source contains one embedded MTP block represented by 15 tensors.

For this output:

• all 15 MTP tensors are byte-identical to the source
• qwen35.nextn_predict_layers remains 1
• all 242 protected / non-quantized tensors are byte-identical
• the final self-contained GGUF loaded successfully with draft-mtp
• the final RTX 3090 smoke generated 128 tokens over two prompts
• 95 draft tokens were generated
• 77 draft tokens were accepted
• smoke acceptance ratio: 0.8105

This acceptance ratio is only a short functional smoke test, not a throughput or representative speculative-decoding benchmark.

The original source was also tested with MTP before quantization and recorded 41 accepted of 44 drafted tokens.

Why the GGUF uses Q4_0 storage

GPTQ-Pro selected the symmetric INT4 group-64 codes/scales. The final GGUF stores those represented weights in llama.cpp-compatible Q4_0 blocks.

This is a representation change, not a second quantization:

• each GPTQ g64 scale is repeated for its two 32-value Q4_0 storage blocks
• every INT4 code is carried over exactly
• every converted matrix was independently dequantized and required exact represented-weight equality
• all 200 quantized decoder matrices passed
• no RTN or second quantizer was applied during export

So the file is best understood as GPTQ-Pro-selected INT4 weights encoded in a llama.cpp GGUF layout, not as a fresh Q4_0 quantization of the source model.

Reload verification

After export:

• all 200 decoder linears were reconstructed from stored packed tensors
• all held-out final hidden states were tensor-exact with the saved quality candidate
• the candidate SHA-256 was rechecked
• the self-contained file loaded on an RTX 3090
• embedded MTP executed successfully

Usage with llama.cpp

Use a llama.cpp build with Qwen3.5 and embedded draft-MTP support.

Example:

./llama-server \
  -m QwenPaw-Flash-9B-heretic-MTP-GPTQPro-INT4-g64.gguf \
  -ngl 99 \
  -c 8192 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --host 127.0.0.1 \
  --port 8080

The final MTP verification used a context of 2048 on an RTX 3090. Larger contexts need their own memory/performance validation. If your build does not support embedded Qwen3.5 MTP, omit the draft-mtp arguments or use a compatible llama.cpp build.

Checksums

source repo:     SC117/QwenPaw-Flash-9B-heretic-MTP-GGUF
source revision: 805a37e8d1277f7db31347283017afb26a04e5a4
source file:     QwenPaw-Flash-9B-heretic-MTP-F16.gguf
source SHA-256:  28039894f5e2388ef9a12f98bd33a9655014f9766f28f5bfe6677ca4e803a77f

output file:     QwenPaw-Flash-9B-heretic-MTP-GPTQPro-INT4-g64.gguf
output SHA-256:  7b2625a843b80186586e1bd895364b83acd10c755b4e7267b9e0ad18cb59fb16
output bytes:    8471015072

Machine-readable results are included in quality-results.json and verification.json.

Limitations

• one calibration seed
• 32 held-out conversations
• bounded/truncated sequences in the controlled experiment
• NLL/perplexity improvement was not statistically established
• source KL is a fidelity metric, not direct coding/tool/factuality/preference accuracy
• no claim that the 30.4% KL reduction generalizes to other models
• MTP acceptance figures are smoke-test evidence, not performance benchmarks
• no throughput speedup claim from this quality experiment
• the source is a heretic / abliterated derivative; this quant does not restore alignment modified upstream

Reproducibility

Quantization code commit:

16bb66e88d8aea424bc3464408f2cad94be5d00e

The experiment used strict numerical checks, fail-closed CUDA-OOM behavior, identical byte budgets, and RTX 3090 GPUs.

Attribution

This is a quantized derivative. Credit for the source model belongs to its original authors and upstream publishers:

• https://huggingface.co/SC117/QwenPaw-Flash-9B-heretic-MTP-GGUF
• https://huggingface.co/agentscope-ai/QwenPaw-Flash-9B
• https://huggingface.co/Qwen/Qwen3.5-9B

Published by groxaxo with measured evidence rather than a generic higher-quality label.