About
weighted/imatrix quants of a Heretic merge of https://huggingface.co/Cyclone-Labs/Twilight-Embrace-31B
Gemma 4 31B dense (60 layers, 262144 context, sliding window 1024). Text GGUFs only — no mmproj. Heretic trial 190 of 200: refusals 59/100, down from a 99/100 baseline, KL divergence 0.0431.
These are i1 (importance-matrix) GGUFs in the mradermacher naming style, converted with llama.cpp on an NVIDIA GB10. Calibration: froggeric/imatrix groups_merged.txt (llama-imatrix -c 512).
Abliteration: p-e-w/heretic v1.4.0. The search ran in NF4 so the 31B weights would fit in 128 GB unified memory. The saved model is a full BF16 merge of that trial, and these GGUFs are quantized from that merge.
Source
Cyclone-Labs/Twilight-Embrace-31B is a MergeKit combine of:
- google/gemma-4-31B-it
- Cyclone-Labs/Eon-Blossom-V2-31B
- UnstableLlama/Love-Aligned-31B
- ReadyArt/Dark-Scarlett-v2.0-31B
Heretic
200 trials, 60 random startup trials, then the optimizer. Scorers were keyword refusal rate (minimize) and KL divergence against the unmodified model on harmless prompts (minimize). Prompts: mlabonne/harmful_behaviors and mlabonne/harmless_alpaca.
Trial 190 was restored and merged. Per-layer direction, weights:
| Component | max weight | max position | min weight | min distance |
|---|---|---|---|---|
| attn.o_proj | 1.50 | 40.33 | 1.10 | 19.49 |
| mlp.down_proj | 1.24 | 41.94 | 0.77 | 3.56 |
Other low-refusal trials that were not selected: trial 90 (65/100, KL 0.0228), trial 100 (68/100, KL 0.0278), trial 113 (69/100, KL 0.0154).
Usage
If you are unsure how to use GGUF files, refer to one of TheBloke's READMEs for more details, including on how to concatenate multi-part files.
Recommended day-to-day quant: Twilight-Embrace-31B-Heretic.i1-Q4_K_M.gguf.
Provided Quants
(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)
Sizes are the on-disk GGUF sizes and are filled in when the files are built.
| Link | Type | Size/GB | Notes |
|---|---|---|---|
| GGUF | imatrix | 0.01 | imatrix file (for creating your own quants) |
| GGUF | i1-IQ3_M | 14.4 | |
| GGUF | i1-IQ4_XS | 16.7 | |
| GGUF | i1-Q4_K_S | 17.8 | optimal size/speed/quality |
| GGUF | i1-Q4_K_M | 18.7 | fast, recommended |
| GGUF | i1-Q5_K_S | 21.3 | |
| GGUF | i1-Q5_K_M | 21.8 | |
| GGUF | i1-Q6_K | 25.2 | practically like static Q6_K |
Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):
And here are Artefact2's thoughts on the matter: https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
Thanks
Source weights by Cyclone-Labs. Heretic by p-e-w. Conversion with llama.cpp.