Dark-Goetia-26B-A4B-LoRA-v4

Atmospheric genre LoRA adapter for bilingual roleplay (English and Russian). Adds a darker, more literary tone to the narrative. Designed exclusively to adjust style and response structure in Dark Fantasy RP. The adapter contains no plots or characters from the training data.

Same target set as v2, different data. v4 trains the identical 115 attention projections with identical rank and alpha. Everything that changed lives in the dataset — see What changed since v2. If you already run v2, this is a drop-in replacement at a similar scale, not a new kind of adapter.

What changed since v2

v2 had one failure mode that only showed up in long sessions: the model would slowly stop obeying the system prompt. Prose quality stayed fine — grammar did not degrade. What degraded was long-horizon instruction following. Somewhere around turn 30–40 the output format quietly drifted away from whatever the character card asked for, and structured blocks started disappearing one at a time.

The cause was in the data. The v2 dataset carried essentially one system prompt and one output format. The model never saw a contrast, so it never learned that format is a variable to be read from the prompt. It learned format as a constant — as part of the style.

v4 attacks that directly. The training set now varies output format across many axes while holding prose style fixed. Style is the constant, format is the variable. If the model sees the same voice delivered in ten different formats, it has to learn to take the format from the prompt rather than from habit.

The stratified eval split is a small change with a large effect on what you can see. With a pooled 5% split, eval loss mostly measures the dominant format and tells you nothing about format generalization — which is the exact thing v4 is trying to fix.

Recommended scale settings

Not yet measured on v4. The numbers below are v2's measured ceilings, carried over as starting points. v4 has the same target set, same rank, and the same alpha, so the mechanics are the same — but the data changed, and the data is what sets the ceiling. Re-measure before you trust these.

Test setup for the v2 numbers: SillyTavern + Marinara's preset + 2000+ token character card.

Free-form prose RP — no strict formatting requirements — IN PROGRESS/TESTING

Cards with strict structured output

If your card requires formatted blocks every turn, the ceiling is considerably lower, and it differs by language.

Russian is measurably more fragile than English at identical adapter strength. If you run bilingual sessions, tune to the Russian number.

Note on scaling: alpha/r = 2 is stored in the GGUF metadata, and llama.cpp multiplies by it on top of your scale. All numbers above assume a loader that respects that. If yours ignores the metadata, halve everything.

Finding your own ceiling in about twenty minutes

The whole point of v4 is a failure mode that does not appear early, so a short test will mislead you.

1. Pick one system prompt with strict output format — named blocks, status trackers, something countable.
2. Run a session out to 30–40 turns. Nothing useful happens at turn 5.
3. Test in Russian. The threshold is lower there, so the Russian number is the one that binds.
4. Step the scale: 0.20, 0.30, 0.40, 0.50.
5. Watch for blocks disappearing, not for grammar getting worse. Grammar stays clean right up until the format is gone. That is what makes this failure easy to miss.
6. Take the last scale that survives 40 turns, subtract 0.05 for margin.

What's actually in the adapter

For anyone merging this or building on it, here is the effective delta ΔW = (B·A)·(alpha/r), Frobenius norms aggregated over layers.

In v2 that worked out to roughly 61% OV / 39% QK by squared norm. The split matters: the OV half carries almost all of the style, while both halves independently contribute to formatting degradation — OV about 3× more readily per unit of weight change. The structured-output ceilings above are set by the OV component alone.

Script to fill in the v4 column

import torch, collections
from safetensors.torch import load_file

sd = load_file("adapter_model.safetensors")
scale = 64 / 32  # alpha / r
acc = collections.defaultdict(float)

for k in sd:
    if not k.endswith("lora_A.weight"):
        continue
    B = sd[k.replace("lora_A", "lora_B")].float()
    A = sd[k].float()
    n2 = ((B @ A) * scale).norm().item() ** 2
    proj = k.split(".lora_A")[0].rsplit(".", 1)[-1]
    acc[proj] += n2

for p, v in sorted(acc.items()):
    print(f"{p:8s} {v ** 0.5:.3f}")
print("OV total", (acc['v_proj'] + acc['o_proj']) ** 0.5)
print("QK total", (acc['q_proj'] + acc['k_proj']) ** 0.5)
print("full    ", sum(acc.values()) ** 0.5)

Training

• Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
• Method: QLoRA 4-bit NF4, double quant, bf16 compute. Attention-only (q/k/v/o_proj), all 30 layers of the text tower. r=32, alpha=64, dropout 0.05, completion-only loss.
• Schedule: 1 epoch, lr 2e-5 cosine, 35 warmup steps, batch 1 × grad accumulation 8, max length 3584, seed 3407. Single A100 80GB.
• Scope: 115 attention projections, not 120 — layers 5, 11, 17, 23 and 29 are global-attention layers with attention_k_eq_v=true and no v_proj at all; the key matrix is reused as the value matrix. 22,978,560 trainable params (0.089%).
• Excluded from training: routed experts, router tensors, shared dense MLP, embeddings, lm_head, and the entire vision tower. Vision layers in Gemma 4 share projection names with text layers, so all target paths are anchored on the language_model.layers. prefix.

Data

Built by a 7-stage pipeline over a dark literary novel, then diluted with external RP data.

1. Scenes. The novel is segmented, speakers attributed, scenes converted to dialogue and validated.
2. Synthetic scenes. New scenes generated in the spirit of the book that are not in the book. This exists because a model trained only on literal book text starts quoting the book instead of generalizing from it.
3. Merge. Book scenes and synthetic scenes are combined into one ShareGPT file.
4. Format augmentation. Every transcript is rewritten across eight format axes — quotation style, dashes, grammatical person, tense, response length, paragraphing, metadata blocks, and so on. Each rewrite is anchored three ways at once: against the original scene, against the canonical transcript, and against explicit format directives.
5. Assembly. External ShareGPT RP data is added as diluent, everything is shuffled and written out.

Format augmentation is a transformation, not a data source — it reformats existing content rather than adding new content. It solves a different problem than synthetic scene generation: synthetic scenes stop the model from quoting, augmentation stops it from ignoring the system prompt.

Three guards run on every rewritten sample, and a sample failing any one of them is dropped:

• Refusal detection. The rewriting model sometimes refuses. A refusal reads as valid text and would poison the set.
• Lexical overlap ≥ 0.55 against the source. Below that, it is a new scene rather than a reformat.
• Distinctive word retention ≥ 0.70. Names, places and rare terms must survive. Russian is inflected, so matching runs on 5-character stems rather than whole words.

Dataset statistics: TBD sessions, TBD RU / TBD EN, median session length TBD tokens, TBD unique system prompts.

Limitations

• 18+. Base model is abliterated, training data includes adult and dark content.
• The improvement is designed, not proven. The v4 dataset was built specifically to fix long-horizon format compliance. Whether it fixed it is a question for measurement, and that measurement is not in this card yet.
• Training context is capped at 3584 tokens. Format collapse in v2 appeared at turn 30–40, well past that. Format augmentation attacks the problem from the data side, but the model still never sees the long horizon during training. This is a real remaining gap.
• Sessions longer than 3584 tokens had their tails truncated, so their final assistant turns were not trained on.
• Single-source style. It comes from one novel and is narrow by construction.
• Base-specific. A LoRA is a delta measured against the weights it was trained on. Applied to a different Gemma-4-26B-A4B checkpoint it usually produces no error, just quiet degradation.
• Inherits Gemma license terms.

Related adapters

• v2 — same 115 targets, single-format data. Superseded by this release, but its scale ceilings are measured and v4's are not yet.
• v3-A — v_proj + o_proj only (55 modules). The style channel without the QK component. Research artifact.
• v3-B — q_proj + k_proj only (60 modules). Research artifact — weak style, published for reproducibility.

A + B = exactly this adapter's target set and parameter budget.