Callsign "Casanova": SubMaroon and Indexnusrefather present to you:
Kitchoon-v3-26B-A4B
Kitchoon trained from scratch on Gemma-4-26B-A4B-it-heretic-antislop — this time with the dense MLP trained alongside OV, on 8192-token windows.
The most stable release of my work. Thinking works! The head is untouched: the base already handles slop at the token level, so instead of a head injection, slop phrases and slop constructions were masked out of the training loss.
The Kitchoon series is designed as an RP- and ERP-oriented model, focusing on character development and featuring a neutral writing style.
What was changed
| Base (heretic-antislop) | Kitchoon-V3 | |
|---|---|---|
Attention OV (v_proj + o_proj) |
stock | LoRA, layers 10–29, 36 targets, baked at 0.5 |
Dense MLP (gate_proj + up_proj + down_proj) |
stock | LoRA, layers 10–20, 33 targets, baked at 0.5 |
lm_head, QK, MoE experts, router, embeddings, vision tower |
stock | identical |
☕ Support
If this model was useful to you, you can support my work with a small donation.
It helps me pay for GPU time and keep releasing new models and datasets.
Gram (TON network): UQB6cmEHWIcuiwsjgO_C0Uz4fyX3Puj09BAB3hEVVpDDcQ7K
Boosty donation: Link
Details
- OV and dense MLP were trained together as one LoRA, but baked with separate strengths (both at 0.5).
- MoE experts and the router were never touched — checked by module name and by matrix shape.
- Baked with stochastic bf16 rounding, so the baked strength matches the tested one.
| Parameter | Value |
|---|---|
| Rank / alpha | OV 32 / 64, MLP 16 / 32 |
| Learning rate | 2.5e-5, cosine |
| Epochs | 1 (754 steps) |
| Context | 8192 tokens |
| Eval loss | 4.32 → 1.41 |
The dataset is sloppier than the base, so training on it naively would bring slop back. The model still sees these spans in context — it just gets no gradient for reproducing them:
- Slop phrases. Stock cliché phrases plus ~400 n-grams that showed up across many different characters in the dataset ("voice dropping to a", "a flicker of", "tilts her head"...).
- Slop constructions. "Not just X, but Y", "It wasn't an X; it was a Y" — the antislop base doesn't catch these.
- "X — or Y?" endings. The final either-or question sentence of a reply.
- Echoing the user. A first sentence that repeats the user's words back.
- Card echo. Text copied verbatim from the character card.
In testing, slop stayed at the level of the clean base.
I would like to express my immense gratitude to Indexnusrefather, with whom I work, for providing the dataset.
DatasetEnglish RP dataset, ~2.8k multi-turn conversations built on character cards.
Cleaned before training:
- removed cards with underage characters and real people;
- fixed ~640 system prompts that instructed the model to write for the user;
- long conversations split into 8192-token chunks instead of being truncated;
- the first and/or last paragraph randomly trimmed in part of the replies, to break up identical openings and hook-question endings;
- character name prefixes (
Name:) stripped from replies.
Using it
[!Note] Recommended settings:
Parameter Value Temperature 0.85Min-P 0.05Repetition penalty / DRY off Thinking works in both modes, including at 5–7K tokens of context. Replies run long (~400 words) — give it a response length of at least 900 tokens.
What to expect
In testing, compared to the base:
- characters stick closer to the card — they pull in side characters, habits and lore from the description on their own;
- more dialogue and a stronger character voice;
- strict formatting rules and structured presets are followed reliably;
- far fewer "X — or Y?" endings and "not just X" constructions.
Known quirks, still there:
- characters sometimes repeat the user's words back in their first line;
- most paragraphs start with a name or "He/She" — this comes from Gemma itself;
- smirk and the em dash have not fully left the building.
Feedback
This is an experiment, and I mostly test it on my own cards, so my coverage is narrow. If you try it, I would really like to know:
- How it holds up in long sessions — scene logic, who is where, who is talking to whom.
- Whether the quirks above bother you in actual play.
- Your sampler settings, if you got clearly better or worse results than with the defaults.
Open a thread in the Community tab, I read all of them.
Credits
Thanks to Tetratype for the antislop base, to coder3101 for the heretic version it is built on, and to Sam Paech for the auto-antislop pipeline. Thanks to the entire 26B-Suite team for their intellectual support.
Special thanks to Naphula and redaihf. You guys are awesome!
And special thanks, of course, to Indexnusrefather for his excellent dataset.