Qwen3.5-2B Swedish Preference & Quality Gate (v4.1)
A single-pass 5-question Swedish quality judge (~58 ms per pair, ~33 ms per response) fine-tuned from Qwen/Qwen3.5-2B.
This model adapts Qwen/Qwen3.5-2B (1.88B parameters, 24 GatedDeltaNet + full-attention layers) to the exact same single-pass discriminator interface and 5 questions (preference, naturalness, quality, natural_swedish, grammatical) as jonasaise/laya-swedish-preference-gate. All 24 backbone layers, the final RMSNorm, and the 7-option decision head (A, B, 1..5; 1.37B trainable parameters) were fine-tuned on a stratified v4.1 sample and calibrated with per-question temperature scaling.
Code, training pipeline, and full evaluation suite: github.com/joenaess/laya-swedish-preference-gate.
Quickstart
pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"
from laya_gate.llm_judge import QwenPreferenceGate
gate = QwenPreferenceGate("jonasaise/qwen3.5-2b-swedish-preference-gate")
# 1. Pairwise preference (or naturalness)
print(gate.predict_preference(prompt, candidate_a, candidate_b))
print(gate.predict_naturalness(candidate_a, candidate_b, prompt=prompt))
# 2. Single-response quality (1..5), natural Swedish, and grammaticality
print(gate.score_quality(prompt, response))
print(gate.score_naturalness(response, prompt=prompt))
print(gate.score_grammar(response))
Length-controlled revision (v4.2b)
Revision v4.2b warm-starts from v4.1 on 800 v4.2b length-controlled counterexample units + 800 stratified v4.1 replay units, lifting short-correct length-test accuracy from 44.0% to 75.9% (overall length_test/preference: not measured, arena_sv preference: 62.0%).
Comparison with Laya-307M (jonasaise/laya-swedish-preference-gate)
Can a small generative LLM (Qwen/Qwen3.5-2B, 1.88B parameters, 24 GatedDeltaNet +
full-attention layers) serve the exact same single-pass discriminator role when adapted with the
same 5 questions (preference, naturalness, quality, natural_swedish, grammatical),
temperature calibration, and v4.1 multi-question training recipe (python -m laya_gate.llm_judge)?
QwenPreferenceGate formats each question with its option criteria (A/B or 1..5),
applies token-level field truncation to 1,024 tokens, disables <think> reasoning, and projects
the final hidden state onto the 7 option vectors in a single forward pass. For fine-tuning on a
single 12 GB GPU, all 24 backbone layers, the final RMSNorm, and the 7-option decision head
(1.37B trainable parameters) are trained with 8-bit PagedAdamW8bit and gradient checkpointing
while freezing the 248k-row tied vocabulary embedding table, on a deterministic stratified
8,000-unit v4.1 sample (400 units per (source, question) group, 800 for fluent/quality),
alongside a matched-budget Laya-307M control trained on the exact same 8,000 examples.
Cost and efficiency
| Model | Parameters | Trainable | Train units | Train time | Peak VRAM | Pair p50 | Single p50 | Throughput (bs=8) |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3.5-2B (zero-shot) | 1882M | 0 | 0 | — | — | 57.7 ms | 33.0 ms | 17.1/s |
| Qwen/Qwen3.5-2B (v4.1 fine-tuned) | 1882M | 1373M | 8,000 | 16.2 min (121.9s/1k) | 7.22 GiB | 57.9 ms | 32.9 ms | 17.2/s |
| Qwen/Qwen3.5-2B (v4.2b length-controlled) | 1882M | 1373M | +1,690 | +3.3 min | 7.22 GiB | 57.7 ms | 32.9 ms | 17.2/s |
| Laya-307M (v4.1 matched 8k) | 307M (322M total) | 322M | 8,000 | 6.7 min (50.4s/1k) | 6.01 GiB | 9.8 ms | 6.0 ms | 119.9/s |
| Laya-307M (v4.1 full) | 307M (322M total) | 322M | 223,521 | 166.0 min (44.6s/1k) | 6.01 GiB | 9.8 ms | 6.0 ms | 119.9/s |
Benchmark accuracy across all configurations and consensus fusion
| Benchmark | Qwen3.5-2B zero-shot | Qwen3.5-2B v4.1 (8k) | Qwen3.5-2B v4.2b (+len) | Laya-307M matched (8k) | Laya-307M v4.1 (224k) | Consensus (Laya ∧ Qwen-FT) |
|---|---|---|---|---|---|---|
In-domain original/preference |
79.1% | 93.6% | 92.7% | 70.9% | 98.8% | 99.4% (94% agree) |
In-domain fluent/preference |
75.8% | 82.1% | 82.4% | 62.9% | 76.6% | 88.8% (76% agree) |
Reasoning-on probe/preference |
73.9% | 82.3% | 84.6% | 62.9% | 79.3% | 88.0% (81% agree) |
DaLAJ-GED naturalness |
73.7% | 86.7% | 86.6% | 60.0% | 88.0% | 93.9% (85% agree) |
DaLAJ-GED grammatical |
73.6% | 75.0% | 74.2% | 53.6% | 81.4% | 84.7% (67% agree) |
ScaLA-sv grammatical acc@0.5 |
not measured | not measured | not measured | not measured | not measured | 81.0% (69% agree) |
GerLangMod-sv naturalness |
94.4% | 96.1% | 96.0% | 64.8% | 95.7% | 99.2% (93% agree) |
BLiMPPI-sv naturalness |
69.5% | 96.6% | 94.9% | 15.3% | 89.8% | 96.4% (93% agree) |
MultiBLiMP-sv naturalness |
84.6% | 99.5% | 100.0% | 46.3% | 97.5% | 100.0% (97% agree) |
WikiEdits-sv naturalness |
51.8% | 58.0% | 56.3% | 49.1% | 56.6% | 60.3% (71% agree) |
SweFAQ preference |
72.9% | 93.9% | 93.2% | 77.2% | 88.6% | 96.9% (88% agree) |
SweFAQ quality top-1 |
24.7% | 40.2% | 41.1% | 16.9% | 23.3% | 23.7% (69% agree) |
Length test (short correct, preference) |
8.6% | 44.0% | 75.9% | 16.9% | 28.6% | 27.3% (61% agree) |
FLORES+ preference+source |
43.6% | 59.5% | 58.3% | 50.2% | 71.9% | 73.6% (67% agree) |
WMT24++ OPUS-MT preference+source |
61.8% | 73.4% | 70.4% | 54.5% | 72.8% | 82.9% (70% agree) |
WMT24++ Kimi-k3 preference+source |
27.7% | 40.7% | 35.6% | 47.6% | 59.8% | 50.5% (59% agree) |
Europarl EN→SV Kimi-k3 preference+source |
20.5% | 45.5% | 41.2% | 43.6% | 75.9% | 67.1% (62% agree) |
Europarl SV→EN→SV Kimi-k3 preference+source |
13.6% | 42.3% | 38.4% | 43.8% | 82.8% | 74.1% (52% agree) |
Chatbot Arena arena_sv (preference) |
63.6% | 61.2% | 62.0% | 62.8% | not measured | 64.4% (67% agree) |
ScaLA-sv grammatical MCC@0.5 |
not measured | not measured | not measured | not measured | not measured | 0.623 (69% agree) |
Takeaways:
- Fine-tuning transforms Qwen3.5-2B as a Swedish judge: full-backbone fine-tuning on the 8,000 stratified
v4.1units lifts Qwen3.5-2B on in-domainoriginal/preference(79.1% → 93.6%),fluent/preference(75.8% → 82.1%), DaLAJ-GED (naturalness73.7% → 86.7%), BLiMPPI-sv (69.5% → 96.6%), SweFAQ (72.9% → 93.9%), and short-correct length-controlled pairs (8.6% → 44.0%), while halving false-positive grammar flagging on clean human references. - Sample efficiency vs full-data encoder specialization: at 8,000 training units, the 1.88B causal LM is far more sample-efficient than the 307M bidirectional encoder trained from scratch on the task head. However, because Laya-307M trains 2.7× faster per unit and runs 5.9× faster at inference (9.8 ms vs 57.9 ms per pair, 119.9 vs 17.2 pairs/s), training Laya-307M on the full 223,521-unit
v4.1recipe surpasses fine-tuned Qwen3.5-2B on DaLAJ-GED (88.0% vs 86.7%), ScaLA-sv acceptability (MCC not measured vs not measured), and every human-vs-MT translation benchmark (FLORES+ 71.9% vs 59.5%, WMT24++ Kimi-k3 59.8% vs 40.7%, Europarl EN→SV Kimi-k3 75.9% vs 45.5%), where generative LLMs retain a strong bias toward machine-generated phrasing. - Standalone vs cascade vs consensus fusion: because the bidirectional encoder (
Laya-307M) and causal LM (Qwen3.5-2B) have complementary inductive biases (linguistic form/anti-MT adequacy vs pretrained instruction helpfulness), on the 69–94% of items where both agree (Laya ∧ Qwen-FT), accuracy is higher than either model's accuracy on all items — a selective-prediction result that abstains on the rest — onfluent/preference(88.8%),probe/preference(88.0%), DaLAJ-GED (93.9%), SweFAQ (96.9%), and ScaLA-sv acceptability (MCC not measured), while standaloneLaya-307Mshould be used for high-volume corpus filtering and source-grounded translation curation. - Length-controlled pairs (
v4.2b) remove Qwen3.5-2B's length bias: warm-starting fromv4.1on 800v4.2blength-controlled counterexample units + 800 replay units liftslength_test/short_correct/preferencefrom 44.0% to 75.9% (overalllength_test/preference: not measured).
License
Apache-2.0 (base model Qwen/Qwen3.5-2B is Apache-2.0).