Gemma-4-26B-A4B-StyleTune-QK-Heretic
Gemma 4 26B-A4B with three edits: refusal directions ablated, lm_head
replaced, attention routing interpolated toward a roleplay finetune.
Released for use as a starting point for further finetunes and merges.
MoE experts, router, embeddings, MLP and the vision tower are bit-for-bit identical to the abliterated body. Verified by tensor comparison.
Recipe
| Component | Source |
|---|---|
| Body | coder3101/gemma-4-26B-A4B-it-heretic (Heretic ARA, layers 10-30, KL 0.0499) |
lm_head |
Gryphe/Gemma-4-26B-A4B-StyleTune-V2, tie_word_embeddings set to false |
q_proj, k_proj |
base + a * (Pantheon-Reasoning-1.1-V2 - gemma-4-26B-A4B-it), 60 tensors |
| Everything else | unchanged from the body |
Alphas: 0.85 on sliding q/k, 0.65 on global q/k.
Global layers are 5, 11, 17, 23, 29. On those layers v_proj does not exist and
k_proj serves as both keys and values, so a non-zero alpha on global k also
injects donor content into the residual stream. Alpha there is lower than on the
sliding layers for that reason, and non-zero so that queries and keys are
rotated together.
StyleTune-V2 trains only lm_head; all 30 transformer layers in it are
bit-identical to vanilla, checked across all 60 QK tensors before merging. That
is why vanilla is used as the task-vector reference even though Pantheon-V2 was
trained on top of StyleTune-V2.
Size of the QK edit
Relative Frobenius norm of the task vector against the base weights, measured before merging:
| Group | mean | max |
|---|---|---|
| sliding_q | 0.00278 | 0.00518 |
| sliding_k | 0.00263 | 0.00474 |
| global_q | 0.00356 | 0.00436 |
| global_k | 0.00286 | 0.00395 |
Per-row rotation between the base weights and the merged weights, in degrees, at the release alphas:
| Group | alpha | mean | max |
|---|---|---|---|
| sliding_q | 0.85 | 0.12 | 3.08 |
| sliding_k | 0.85 | 0.12 | 5.87 |
| global_q | 0.65 | 0.12 | 1.72 |
| global_k | 0.65 | 0.09 | 9.53 |
The rotation is concentrated in a small number of rows: maximum over mean runs from about 14 on global_q to about 106 on global_k.
Gemma applies RMSNorm after q_proj and k_proj, so uniform rescaling of these
weights does not affect attention. Only rotation matters, which is why the angle
is reported alongside the norm.
What the QK step changes
The comparison below isolates the QK step. Both arms are the same build (abliterated body, StyleTune head) and differ only in the 60 QK tensors. Effects shared with vanilla Gemma 4, and effects produced by the head transplant, are excluded.
Four prompts, two light and two dark, greedy decoding, two modes each, 2500 token budget. All generations terminated normally. Full dump included in the repo.
Prompt parsing improves. The merge more often acts on the specific situation stated in the prompt, including its participants and their actions. On a prompt about discovering that a trusted party has already signed against the character, the pre-QK build writes about the feeling of betrayal; the merge identifies the document, the mismatched handwriting and the prior meeting. On a prompt about a neighbour calling over a fence, the merge stages the neighbour's approach and the call; the pre-QK build has the neighbour present but not calling.
Lexical diversity drops slightly. Type/token ratio 0.412 before, 0.392 after, over about 2000 words per arm. Small enough that it may be noise at this sample size, but it does not move upward.
Length is unchanged. Mean prose length 1443 chars before, 1487 after. Mean word length 4.49 before, 4.52 after.
Thinking-block length moved in opposite directions on two different prompt sets and is therefore not reported.