UPDATE 2026-07-20: Download & use Google's new chat template from here for better speed and accurancy. See Unsloth's post for details.
WARNING: Created with heavy LLM assistance (Zoo Code + DeepSeek-V4-Flash). Use at your own discretion.
Cosplayed Frankenstein and grafted/"merged" llmfan46's abliterated tensors onto Unsloth's lossless Q4_0 quant. Should yield better accurancy and refusal rate than a naive abliterated Q4_0 quant.
This repo contains two variants: "UDmerge-Q4_K_XL" have the abliterated tensors (blk.N.attn_output.weight, where N is 11 to 22 inclusive) quantized to Q4_0, while "UDmerge-Q4_K_XXL" quantizes them to Q8_0. The latter improves refusal rate by a lot, while basically not affecting TG speed (your mileage may vary).
Use Unsloth's mmproj and mtp GGUF files for multimodal and MTP support.
| - | QAT BF16 | QAT Q4_0 | QAT Q4_K_M | QAT UDmerge-Q4_K_XL | QAT UDmerge-Q4_K_XXL | - | PTQ BF16 | PTQ Q4_0* | PTQ Q4_K_M |
|---|---|---|---|---|---|---|---|---|---|
| Size (GB) | 51.6** | 16.0 | 16.8 | 14.2 | 14.3 | - | 51.6** | 14.5 | 16.8 |
| PPL | 4.945 | 4.804 | 5.246 | 5.236 | 5.101 | - | 5.279 | 10.17 | 8.177 |
| KLD | 0.0000 | 0.1158 | 0.1076 | 0.0448 | 0.0291 | - | 0.0000 | 0.7388 | 0.6130 |
| Refusal | 17% | 19% | 25% | 33% | 16% | - | 28% | 27% | 26% |
| MMLU-val | 80.86% | 80.60% | 80.73% | 81.38% | 81.06% | - | 80.54% | 79.49% | 79.88% |
| MMLU-val %flips | 0.00% | 3.53% | 3.79% | 3.66% | 0.98% | - | 0.00% | 5.23% | 5.88% |
| MMLU-val %allflips | 0.00% | 4.31% | 4.90% | 4.83% | 1.76% | - | 0.00% | 7.38% | 7.51% |
Legend:
*: Quant made with importance matrix ("imatrix"), results may be unreliable**: Includes multimodal ("mmproj") weights, which is ~1.2GB in BF16
PPL and KLD are tested on the same dataset as Heretic, i.e. the first 100 questions in the mlabonne/harmless_alpaca dataset's test split. Note that the dataset is processed differently, thus the numbers here are only meaningful for comparsions in this table, not with other models.
Refusal rates are also tested on the same dataset as Heretic, i.e. the first 100 questions in the mlabonne/harmful_behaviors dataset's test split. The test script, however, is adapted from Heretic to support testing needs. Note that the original author claimed 11% refusal rate for 31B and 26B-A4B models, and 6%~7% for 12B, which is not reproduced here; this is probably due to test method differences, but please take the numbers here with a grain of salt.
"MMLU-val" refers to zero-shot testing on the cais/mmlu dataset's validation split (1531 questions). All tests are done once with temperature 0.0 and reasoning off. MTP is not enabled during testing. See the test script and raw data for details.
"%flips" and "%allflips" refer to the percentage of changed answers compared to BF16 models, measured as by the paper Accuracy is Not All You Need (arXiv:2407.09141). "%flips" is the percentage of "right-to-wrong" and "wrong-to-right" changes, while "%allflips" is the percentage of all changed answers.
More test results| - | QAT BF16 | QAT Q4_0 | QAT Q4_K_M | QAT UDmerge-Q4_K_XL | QAT UDmerge-Q4_K_XXL | - | PTQ BF16 | PTQ Q4_0* | PTQ Q4_K_M |
|---|---|---|---|---|---|---|---|---|---|
| MMLU-val-v1 | 78.84% | 77.73% | 77.86% | 77.07% | 78.90% | - | 79.29% | 79.82% | 76.49% |
| MMLU-val-v1 %nulls | 3.79% | 4.25% | 5.94% | 7.32% | 3.98% | - | 3.53% | 1.70% | 7.84% |
| MMLU-val-v1 %flips | 0.00% | 3.20% | 4.51% | 5.03% | 1.11% | - | 0.00% | 5.23% | 7.12% |
| MMLU-val-v1 %allflips | 0.00% | 4.31% | 5.68% | 6.47% | 2.02% | - | 0.00% | 7.45% | 9.54% |
"MMLU-val-v1" numbers are "MMLU-val" test done by an older version of the test script, which does not enforce grammar constraints on answer format. These number are less representative than the ones given above (?), and are kept here for reference purposes only. "%nulls" here refers to the percentage of "null answers", i.e. answers that are not in the required format, thus not parsable.
More information, including test scripts and raw test data, will be released soon.