Rumeni-26B-A4B
A model for roleplay in Russian language. A LoRA on attention OV and the dense MLP, trained on a Russian RP dataset and baked into my base StyleTune-QK-Heretic.
Keep your usual English cards.
What was changed
| Base (StyleTune-QK-Heretic) | Rumeni-26B-A4B | |
|---|---|---|
Attention OV (v_proj + o_proj) |
stock | LoRA, layers 10–29, 36 targets, baked at 0.5 |
Dense MLP (gate_proj, up_proj, down_proj) |
stock | LoRA, layers 0–29, 90 targets, baked at 0.5 |
QK, MoE experts, router, lm_head, embeddings, vision tower |
stock | identical |
Details
- One LoRA, two groups of modules, trained together and baked with separate scales.
- Baked in fp32 with stochastic rounding to bf16. Plain rounding lost about 30% of a small LoRA delta in my earlier builds; here the scale measured in the weights matches the nominal one within 3%.
- The adapter itself is in
lora/in this repo.
I would like to express my deep gratitude to DefaultDF for providing the dataset.
LoRA details| Parameter | Value |
|---|---|
| Rank / alpha | OV 32 / 64, MLP 64 / 128 |
| Learning rate | 2e-5, cosine |
| Epochs | 1 (1198 steps) |
| Context | 6144 tokens |
| Eval loss (final) | 1.13 |
OV decides what gets carried over from the context. The MLP holds wording and word forms, and a language shift needs both. So this time the dense MLP, the one every token passes through, was trained on all 30 layers. Routed experts and the router were left alone.
The router reads the same residual stream, so expert choice shifts a little: at the release scale about 90% of expert picks match the base.
Using it
[!Note] Recommended settings:
Parameter Value Temperature 1.0Min-P 0.05Put this line at the end of the system prompt (or in Author's Note):
[Пиши все ответы только на русском языке: и повествование, и прямую речь.]Thinking off — the training data had no reasoning blocks, and thinking mode was not tested.
What to expect
In testing on English cards, compared to the base:
- Russian holds: Latin letters stay under 1% of the text, nearly all of it names and brands;
- replies come out around 250 words and end on their own, while the base often ran past 600 tokens;
- two-character cards stay apart: each character keeps their own voice, and formal «вы» is kept where the card sets it.
Known quirks, still there:
- replies like to end on a question, more often on Russian-language cards;
- characters sometimes repeat the user's question back before answering;
- an occasional English word slips through (sweetheart), and agreement errors are rare but present.
☕ Support
If this model was useful to you, you can support my work with a small donation.
It helps me pay for GPU time and keep releasing new models and datasets.
Gram (TON network): UQB6cmEHWIcuiwsjgO_C0Uz4fyX3Puj09BAB3hEVVpDDcQ7K
Boosty donation: Link
Feedback
This is an experiment, and I mostly test it on my own cards, so my coverage is narrow. If you try it, I would really like to know:
- How the Russian reads to a native speaker: word forms, gender agreement, calques.
- How it holds up in long sessions — scene logic, who is where, who is talking to whom.
- Your sampler settings, if you got clearly better or worse results than with the defaults.
Open a thread in the Community tab, I read all of them.
Credits
Thanks to Gryphe for StyleTune and Pantheon, which gave the base its head and QK. Also thanks to the entire 26B-Suite team for their intellectual support.
Special thanks to Naphula and redaihf. You guys are awesome!