Qwen2.5-Math-1.5B-Instruct: French (v3)
Qwen2.5-Math-1.5B-Instruct, fine-tuned to understand math word problems written in French. The base model is strong in English but weak in French. This version closes about 60% of that gap, with English unchanged.
Results (greedy decoding)
| Model | MGSM-fr (250) | French dev set (250)* | MGSM-en (250) |
|---|---|---|---|
| Qwen2.5-Math-1.5B-Instruct | 38.8% | 36.0% | 86.0% |
| v1 (2.7k examples, opus-mt only) | 57.6% | 57.2% | 85.6% |
| v2 (13.6k examples, opus-mt + Luth) | 67.2% | 65.6% | 82.8% |
| v3, this model (v2 + 1 epoch with 2k English mixed in) | 68.0% | 65.6% | 86.4% |
With majority voting (8 sampled answers per question, temperature 0.7, the most common answer wins), measured on this model:
| MGSM-fr | French dev set | |
|---|---|---|
| greedy | 68.0% | 65.6% |
| single sample | 67.0% | 65.9% |
| vote@8 | 74.0% | 72.8% |
| pass@8 (right answer among the 8) | 83.6% | 86.8% |
*The dev set is 250 GSM8K test questions that are not in MGSM, translated to French. Choices between versions are made on it, so MGSM-fr stays an honest test score. Noise is about ±3 points at n=250. v2 dipped 2.8 points in English. v3 fixes that, back to the base model's level, with no French change (both differences are within noise).
For comparison, kurakurai/Luth-2-2B, a larger French-tuned model, scores 77.2% on MGSM-fr under the same eval with a 2048-token budget.
Side challenge: catch Luth-2-2B
I'm trying to fine-tune this 1.5B model to tie or beat kurakurai/Luth-2-2B, a larger model built for French, on French math. Same eval script, greedy decoding:
| This model (1.5B) | Luth-2-2B (2B) | ||
|---|---|---|---|
| MGSM-fr, greedy | 68.0% | 77.2% | Luth ahead by 9.2 |
| MGSM-fr, this model with vote@8 | 74.0% | 77.2% (greedy) | gap 3.2, not like-for-like |
| French MATH-500 (first 200, harder) | **65.0%**¹ | 62.0% | tie (within ±3) |
| MGSM-en | 86.4% | 80.4% | this model ahead |
¹ measured on v1.
Luth starts from a newer, larger, multilingual base. This model is a math specialist that had to learn to read French. Its math is already there: 86% in English and 84% pass@8 in French. The remaining gap is French comprehension. Next steps: GRPO on French questions, more natural-French training data, then a larger LoRA. Progress is tracked in the GitHub repo.
How it was made
Cross-lingual self-distillation, scaled up from v1:
- All 7,473 GSM8K train questions were translated to French twice: once with
Helsinki-NLP/opus-mt-en-fr(literal) and once withkurakurai/Luth-2-2B(natural French). Translations that dropped or changed a number were discarded. - The base model solved the original English questions, and only correct solutions were kept (7,154 of 7,473).
- LoRA SFT on French question → the model's own correct English solution: 13,644 examples (6,720 opus-mt + 6,924 Luth), r=32 on all attention and MLP projections, 1 epoch, lr 1e-4, then merged.
- v3: one more epoch from v2 at lr 5e-5 on the same data plus 2,000 English examples (English question → own solution), to undo v2's small English drop.
The targets are the model's own solutions, so it learns to understand French without changing how it reasons. MGSM's test problems come from GSM8K's test split, so there is no train/test overlap.
Trained on one RTX 5060 Ti (16 GB): 50 minutes for v2, plus 57 minutes for v3. Code, evals and negative results: github.com/GAMINGZIYADX/small-math-tuning. v1 and v2 are still available in this repo's commit history.
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
name = "GAMINGZI/qwen2.5-math-1.5b-french"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="auto")
msgs = [
{"role": "system", "content": "Please reason step by step, and put your final answer within \\boxed{}."},
{"role": "user", "content": "Un magasin vend des stylos à 3 € pièce. Léa en achète 4 et paie avec un billet de 20 €. "
"Combien d'argent lui rend-on ?"},
]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
Majority voting (recommended for best accuracy)
import collections, re
def last_boxed(text):
i = text.rfind("\\boxed")
if i < 0: return None
j, depth = text.find("{", i), 0
for k in range(j, len(text)):
depth += {"{": 1, "}": -1}.get(text[k], 0)
if depth == 0: return text[j + 1:k].strip()
return None
out = model.generate(**enc, max_new_tokens=1024, do_sample=True, temperature=0.7, top_p=0.95, num_return_sequences=8)
answers = [last_boxed(t) for t in tok.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)]
votes = collections.Counter(a for a in answers if a)
print(votes.most_common(1)[0][0] if votes else None)
It costs 8× the compute of one greedy answer, and adds about 6–7 points on French grade-school problems.
Limitations
- It reads French but answers in English, because it was trained to reproduce its own English reasoning.
- The training questions are machine-translated, even though half of them come from a fluent model.
- No measurable gain on harder French math. v1 scored 65.0% on French MATH-500 (first 200, translated), against 62.5% for the base model, which is within noise. v3 has not been measured there yet.
- It is still about 15 points weaker in French than in English on grade-school problems.