Laya Swedish Preference Gate (v4.1)
A ~10 ms judge for Swedish text. One forward pass of a fine-tuned Laya decision model (mmBERT-base backbone) answers one of five questions, as a "System 1" filter in front of a slow LLM judge.
| Question | Input | Output |
|---|---|---|
preference |
prompt (or English source) + two responses | calibrated P(response A is better) |
naturalness |
two responses (prompt optional) | P(A is more natural, idiomatic Swedish) |
quality |
prompt + one response | 1–5 score |
natural_swedish |
one response | P(natural Swedish vs translationese) |
grammatical (new) |
one text | P(grammatically correct); compare two texts, see limitations |
Earlier releases:
revision="v4",revision="v3",revision="v2",revision="v1".
Code, the full ablation and every number below: github.com/joenaess/laya-swedish-preference-gate.
Training and evaluation pairs: jonasaise/laya-swedish-gate-v4-data.
What changed from v3
v4 and v4.1 add training data chosen in a controlled ablation (same recipe, seed and hyperparameters; only the data differs; all tests held out):
- DaLAJ-GED (Superlim 2): learner sentences vs the same sentence corrected, for
naturalnessand the newgrammaticalquestion. - SweFAQ (Superlim 2): a question's own answer vs other answers from the same FAQ page,
for
preference. - Europarl (Språkbanken): professional Swedish translations vs Kimi-k3 machine translations of English speeches, plus adequacy negatives (another paragraph's translation, a half translation, a word-shuffled translation, the untranslated English).
- v4.1 only: correct text from four registers against corrupted copies (a word deleted or
two neighbours swapped), and wrong-source negatives from the adjacent paragraph of the same
debate, to reduce
grammaticalover-flagging and fluent-but-wrong translations winning.
| Held-out test (share where the reference wins) | v3 | v4 | v4.1 |
|---|---|---|---|
Fluent-vs-fluent probe, preference |
79.9% | 78.6% | 79.3% |
DPO validation, preference |
98.9% | 98.7% | 98.8% |
liu-nlp native vs OPUS-MT, naturalness |
44.3% | 52.2% | 51.0% |
FLORES+ human vs OPUS-MT, preference with source |
48.5% | 73.8% | 71.9% |
WMT24++ human vs OPUS-MT, preference with source |
54.3% | 74.6% | 72.8% |
WMT24++ human vs Kimi-k3, preference with source |
43.1% | 58.9% | 59.8% |
DaLAJ corrected vs learner, naturalness |
55.9% | 89.7% | 88.0% |
GerLangMod-sv original vs verb misplaced, naturalness |
83.3% | 95.5% | 95.7% |
SweFAQ own answer vs other, preference |
60.3% | 88.9% | 88.6% |
BLiMPPI-sv polarity items, naturalness |
40.7% | 86.4% | 89.8% |
MultiBLiMP-sv agreement, naturalness |
83.1% | 97.0% | 97.5% |
| ScaLA-sv MCC (EuroEval's metric), single-response | 0.146 | 0.341 | 0.466 |
Calibration: preference T = 1.674, naturalness T = 0.485, quality T = 1.226, natural_swedish T = 1.495, grammatical T = 1.427. ECE on the DPO validation pairs 0.0164.
A trap we hit on the way. Human-vs-MT pairs alone teach a gate that "the candidate least like the source wins": the intermediate arms preferred a fluent translation of the wrong sentence, half translations and nonsense. The adequacy negatives fix it (FLORES+ devtest, English source in the prompt, share where the reference wins):
| The reference against … | v3 | without adequacy negatives | v4 | v4.1 |
|---|---|---|---|---|
| the reference of a different sentence | 91.0% | 0.0% | 78.0% | 89.5% |
| LFM2-350M's translation (pilot's lowest chrF) | 97.0% | 3.5% | 98.0% | 97.5% |
| only the first half of the reference | 72.0% | 7.5% | 100.0% | 100.0% |
| the untranslated English | 20.5% | 100.0% | 100.0% | 100.0% |
the untranslated English (naturalness, no prompt) |
87.5% | 38.5% | 99.5% | 99.5% |
On 5600 translations by 28 LLMs, v4.1's translation score agrees with chrF at Spearman 0.38 per translation (v3: 0.30).
Pilot: a quick proxy for EuroEval Swedish
Scoring 100 Swedish answers per model with quality and averaging, over 28 open LLMs with
Swedish results on the EuroEval leaderboard: Spearman with the Swedish
rank 0.66 [0.33, 0.85],
0.63 [0.25, 0.81] after controlling for
model size. A pilot (models up to 11B parameters, 6 of them in 4-bit), not a calibrated
predictor.
Quickstart
uv pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"
from laya_gate.predictor import SwedishPreferenceGate
gate = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate")
r = gate.predict_preference(
prompt="Hur fungerar fotosyntesen?",
chosen_candidate="Fotosyntesen är den process där växter omvandlar ljus till energi.",
rejected_candidate="Det är när solen lyser på träd och gör magi.",
)
# {"preference": "candidate_a", "p_chosen": <float>, "is_valid_margin": <bool>, ...}
# translation quality: put the English source in the prompt
t = gate.predict_preference(
prompt="Översätt följande text till svenska:
The meeting was postponed.",
chosen_candidate="Mötet sköts upp.",
rejected_candidate="Mötet ställdes in.",
)
s = gate.score_quality(prompt="Hur fungerar fotosyntesen?", response="Växter gör mat av solljus.")
g = gate.score_grammar("Människor är snälla där och alla varor är billiga.")
a, b = (gate.score_grammar(x)["p_correct"] for x in ("Hon har bott här.", "Hon har bo här."))
Limitations
grammaticalover-flags as a stand-alone verdict. At 0.5 it marks 25.2% of fluent LLM answers and 47.9% of WMT24++ references as incorrect (it was trained on single learner sentences). Compare two versions of a text instead. On lowercased text without punctuation (GerLangMod-sv) it is at chance (86.3%); v4 was at 47.8%.- Wrong-content translations still win sometimes: v4 prefers a fluent translation of a different sentence in 10.5% of FLORES+ pairs (v3: 9.0%).
- Answer preference largely tracks length. On 129 human Chatbot
Arena votes on Swedish answers,
preferenceagrees with people in 58.9% of battles, close to "the longer answer wins" (64.3%); Kimi-k3 reaches 74.8%. Use the gate for Swedish form (grammar, translationese, translation adequacy, relevance), not to pick between two fluent answers. - Pairwise preference, not fact-checking; 1,024-token window; one seed per training arm.
- Kimi-k3 is not a neutral reference for its own output: on WMT24++ it prefers the human translation over OPUS-MT in 81.0% of pairs but over its own translation in only 24.5%.
Small LLM-as-a-judge comparison (Qwen/Qwen3.5-2B)
Can a small generative LLM (Qwen/Qwen3.5-2B, 1.88B parameters, 24 GatedDeltaNet +
full-attention layers) serve the exact same single-pass discriminator role when adapted with the
same 5 questions (preference, naturalness, quality, natural_swedish, grammatical),
temperature calibration, and v4.1 multi-question training recipe (python -m laya_gate.llm_judge)?
Published on the Hugging Face Hub: jonasaise/qwen3.5-2b-swedish-preference-gate.
QwenPreferenceGate formats each question with its option criteria (A/B or 1..5),
applies token-level field truncation to 1,024 tokens, disables <think> reasoning, and projects
the final hidden state onto the 7 option vectors in a single forward pass. For fine-tuning on a
single 12 GB GPU, all 24 backbone layers, the final RMSNorm, and the 7-option decision head
(1.37B trainable parameters) are trained with 8-bit PagedAdamW8bit and gradient checkpointing
while freezing the 248k-row tied vocabulary embedding table, on a deterministic stratified
8,000-unit v4.1 sample (400 units per (source, question) group, 800 for fluent/quality),
alongside a matched-budget Laya-307M control trained on the exact same 8,000 examples.
Cost and efficiency
| Model | Parameters | Trainable | Train units | Train time | Peak VRAM | Pair p50 | Single p50 | Throughput (bs=8) |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3.5-2B (zero-shot) | 1882M | 0 | 0 | — | — | 57.7 ms | 33.0 ms | 17.1/s |
| Qwen/Qwen3.5-2B (v4.1 fine-tuned) | 1882M | 1373M | 8,000 | 16.2 min (121.9s/1k) | 7.22 GiB | 57.9 ms | 32.9 ms | 17.2/s |
| Qwen/Qwen3.5-2B (v4.2b length-controlled) | 1882M | 1373M | +1,690 | +3.3 min | 7.22 GiB | 57.7 ms | 32.9 ms | 17.2/s |
| Laya-307M (v4.1 matched 8k) | 307M (322M total) | 322M | 8,000 | 6.7 min (50.4s/1k) | 6.01 GiB | 9.8 ms | 6.0 ms | 119.9/s |
| Laya-307M (v4.1 full) | 307M (322M total) | 322M | 223,521 | 166.0 min (44.6s/1k) | 6.01 GiB | 9.8 ms | 6.0 ms | 119.9/s |
Benchmark accuracy across all configurations and consensus fusion
| Benchmark | Qwen3.5-2B zero-shot | Qwen3.5-2B v4.1 (8k) | Qwen3.5-2B v4.2b (+len) | Laya-307M matched (8k) | Laya-307M v4.1 (224k) | Consensus (Laya ∧ Qwen-FT) |
|---|---|---|---|---|---|---|
In-domain original/preference |
79.1% | 93.6% | 92.7% | 70.9% | 98.8% | 99.4% (94% agree) |
In-domain fluent/preference |
75.8% | 82.1% | 82.4% | 62.9% | 76.6% | 88.8% (76% agree) |
Reasoning-on probe/preference |
73.9% | 82.3% | 84.6% | 62.9% | 79.3% | 88.0% (81% agree) |
DaLAJ-GED naturalness |
73.7% | 86.7% | 86.6% | 60.0% | 88.0% | 93.9% (85% agree) |
DaLAJ-GED grammatical |
73.6% | 75.0% | 74.2% | 53.6% | 81.4% | 84.7% (67% agree) |
ScaLA-sv grammatical acc@0.5 |
not measured | not measured | not measured | not measured | not measured | 81.0% (69% agree) |
GerLangMod-sv naturalness |
94.4% | 96.1% | 96.0% | 64.8% | 95.7% | 99.2% (93% agree) |
BLiMPPI-sv naturalness |
69.5% | 96.6% | 94.9% | 15.3% | 89.8% | 96.4% (93% agree) |
MultiBLiMP-sv naturalness |
84.6% | 99.5% | 100.0% | 46.3% | 97.5% | 100.0% (97% agree) |
WikiEdits-sv naturalness |
51.8% | 58.0% | 56.3% | 49.1% | 56.6% | 60.3% (71% agree) |
SweFAQ preference |
72.9% | 93.9% | 93.2% | 77.2% | 88.6% | 96.9% (88% agree) |
SweFAQ quality top-1 |
24.7% | 40.2% | 41.1% | 16.9% | 23.3% | 23.7% (69% agree) |
Length test (short correct, preference) |
8.6% | 44.0% | 75.9% | 16.9% | 28.6% | 27.3% (61% agree) |
FLORES+ preference+source |
43.6% | 59.5% | 58.3% | 50.2% | 71.9% | 73.6% (67% agree) |
WMT24++ OPUS-MT preference+source |
61.8% | 73.4% | 70.4% | 54.5% | 72.8% | 82.9% (70% agree) |
WMT24++ Kimi-k3 preference+source |
27.7% | 40.7% | 35.6% | 47.6% | 59.8% | 50.5% (59% agree) |
Europarl EN→SV Kimi-k3 preference+source |
20.5% | 45.5% | 41.2% | 43.6% | 75.9% | 67.1% (62% agree) |
Europarl SV→EN→SV Kimi-k3 preference+source |
13.6% | 42.3% | 38.4% | 43.8% | 82.8% | 74.1% (52% agree) |
Chatbot Arena arena_sv (preference) |
63.6% | 61.2% | 62.0% | 62.8% | 58.9% | 64.4% (67% agree) |
ScaLA-sv grammatical MCC@0.5 |
0.307 | 0.391 | 0.323 | -0.002 | 0.466 | 0.623 (69% agree) |
Takeaways:
- Fine-tuning transforms Qwen3.5-2B as a Swedish judge: full-backbone fine-tuning on the 8,000 stratified
v4.1units lifts Qwen3.5-2B on in-domainoriginal/preference(79.1% → 93.6%),fluent/preference(75.8% → 82.1%), DaLAJ-GED (naturalness73.7% → 86.7%), BLiMPPI-sv (69.5% → 96.6%), SweFAQ (72.9% → 93.9%), and short-correct length-controlled pairs (8.6% → 44.0%), while halving false-positive grammar flagging on clean human references. - Sample efficiency vs full-data encoder specialization: at 8,000 training units, the 1.88B causal LM is far more sample-efficient than the 307M bidirectional encoder trained from scratch on the task head. However, because Laya-307M trains 2.7× faster per unit and runs 5.9× faster at inference (9.8 ms vs 57.9 ms per pair, 119.9 vs 17.2 pairs/s), training Laya-307M on the full 223,521-unit
v4.1recipe surpasses fine-tuned Qwen3.5-2B on DaLAJ-GED (88.0% vs 86.7%), ScaLA-sv acceptability (MCC 0.466 vs 0.391), and every human-vs-MT translation benchmark (FLORES+ 71.9% vs 59.5%, WMT24++ Kimi-k3 59.8% vs 40.7%, Europarl EN→SV Kimi-k3 75.9% vs 45.5%), where generative LLMs retain a strong bias toward machine-generated phrasing. - Standalone vs cascade vs consensus fusion: because the bidirectional encoder (
Laya-307M) and causal LM (Qwen3.5-2B) have complementary inductive biases (linguistic form/anti-MT adequacy vs pretrained instruction helpfulness), on the 69–94% of items where both agree (Laya ∧ Qwen-FT), accuracy is higher than either model's accuracy on all items — a selective-prediction result that abstains on the rest — onfluent/preference(88.8%),probe/preference(88.0%), DaLAJ-GED (93.9%), SweFAQ (96.9%), and ScaLA-sv acceptability (MCC 0.623), while standaloneLaya-307Mshould be used for high-volume corpus filtering and source-grounded translation curation. - Length-controlled pairs (
v4.2b) remove Qwen3.5-2B's length bias: warm-starting fromv4.1on 800v4.2blength-controlled counterexample units + 800 replay units liftslength_test/short_correct/preferencefrom 44.0% to 75.9% (overalllength_test/preference: not measured).
Training data and licenses
Weights: Apache-2.0. Base model convaiinnovations/laya: apache-2.0.
Training data:
jonasaise/dpo-swedish-interpreted(mit) andargilla/ultrafeedback-binarized-preferences-cleaned(mit).- Europarl Swedish-English: Koehn (2005), Europarl: A Parallel Corpus for Statistical Machine Translation, MT Summit X; enriched and distributed by Språkbanken, https://doi.org/10.23695/mdp4-2p72, CC BY 4.0.
- DaLAJ-GED (CC BY 4.0) and SweFAQ (CC BY-SA 4.0) from Superlim 2
(
sbx/superlim-2), Språkbanken. - Machine translations by
Helsinki-NLP/opus-mt-en-sv(evaluation only) and Kimi-k3.
Evaluation only: liu-nlp/translationese-opensubtitles (Apache-2.0), FLORES+, WMT24++
(Apache-2.0), ScaLA-sv (mteb/scala_sv_classification), GerLangMod (Apache-2.0), BLiMPPI
(arXiv:2610.13607; Swedish subset from UD SweLL/Talbanken CC BY-SA 4.0 and LinES CC BY-NC-SA
4.0), MultiBLiMP (jumelet/multiblimp, CC BY 4.0).
@software{laya_swedish_preference_gate,
title = {Laya Swedish Preference Gate},
author = {Lind, Jonas},
year = {2026},
url = {https://huggingface.co/jonasaise/laya-swedish-preference-gate}
}