🧠 Ministral 3 3B - Qwen 3.8 Max Distillation
A fine-tuned Ministral 3 3B Reasoning model trained on a curated subset of Qwen 3.8 Max distillation data. The goal is to bring the knowledge and response patterns captured in the teacher-generated dataset into a compact model that's practical to experiment with and run.
This model uses unsloth/Ministral-3-3B-Reasoning-2512 as its base and saidutta69/qwen3.8-max-distillation-50k-clean as its training data source.
✨ Model at a glance
| Detail | Value |
|---|---|
| Base model | unsloth/Ministral-3-3B-Reasoning-2512 |
| Teacher/data source | Qwen 3.8 Max distillation dataset |
| Training dataset | saidutta69/qwen3.8-max-distillation-50k-clean |
| Training examples | 8,000 |
| Evaluation examples | 400 |
| Response-length filter | Maximum 6,000 characters |
| Evaluation split | Randomly sampled |
| Model size | 3B-class |
| Primary task | Text generation and instruction following |
🎯 What is this model for?
This fine-tune explores how much useful behavior from a larger teacher-model dataset can be transferred to a smaller reasoning-oriented model.
Potential use cases include:
- 💬 General conversational assistance and instruction following
- 🧩 Reasoning-oriented prompts and multi-step problem solving
- ✍️ Writing, rewriting, and summarization
- 🛠️ Experimentation with synthetic-data distillation and compact language models
These are intended use cases, not a guarantee of performance. The model's actual strengths will depend on the training procedure, data quality, and evaluation results.
📚 Training data
The model was fine-tuned using a filtered subset of saidutta69/qwen3.8-max-distillation-50k-clean, a dataset associated with Qwen 3.8 Max distillation.
The dataset was prepared as follows:
- Response-length filtering: Examples with responses longer than 6,000 characters were excluded.
- Training subset: 8,000 entries were randomly sampled for training.
- Evaluation subset: 400 entries were randomly sampled for evaluation.
This gives a total of 8,400 selected examples across the training and evaluation sets.
The character limit helps keep exceptionally long responses out of the fine-tuning set. It is a character-based filter, not a token limit, so the corresponding token lengths can vary.
⚙️ Training approach
The base model is unsloth/Ministral-3-3B-Reasoning-2512, with the Qwen distillation dataset providing the examples used for LoRA fine-tuning over a single epoch.
The intent is to adapt the smaller model to patterns found in the teacher-generated responses while retaining the practical advantages of a compact model.
| Parameter | Value |
|---|---|
| Base model | unsloth/Ministral-3-3B-Reasoning-2512 |
| Fine-tuning method | LoRA, 16-bit |
| Training examples | 8,000 |
| Evaluation examples | 400 |
| Epochs | 1 |
| Maximum context length | 8,192 tokens |
| Learning rate | 2e-4 |
LoRA rank (r) |
16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.02 |
| Optimizer | AdamW 8-bit |
| Learning rate scheduler | Linear |
| Weight decay | 0.002 |
| Batch configuration | 4 × 8 |
📊 Training results
| Metric | Result |
|---|---|
| Training loss | 0.513 |
| Evaluation loss | 0.517 |
The evaluation loss ended up lower than the training loss. These values are useful as training diagnostics, but they aren't direct measures of reasoning quality and don't, by themselves, establish an improvement over the original model.
Benchmark comparisons and qualitative evaluations on unseen problems would help determine whether the fine-tune improves reasoning, coding, or mathematical performance.
💬 Usage
Required base Mistral Reasoning system prompt (always set this):
# HOW YOU SHOULD THINK AND ANSWER
First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.
Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.
You can append additional instructions to the base system prompt.
⚠️ Limitations
- Distillation is not a guarantee of equivalence. A 3B-class model will not necessarily reproduce the teacher's capabilities or outputs.
- Synthetic-data limitations carry over. Errors, biases, and stylistic quirks in the source responses may be learned by the fine-tune.
- Reasoning can be unreliable. Fluent explanations may still contain incorrect conclusions or unsupported claims.
- Evaluation coverage is limited. A 400-example random sample may not adequately represent every task or domain.
- Language coverage is unspecified. English is listed as the intended primary language; broader language performance has not been established.
This model should not be relied upon as the sole source for high-stakes decisions.
📜 License and attribution
The base model and training dataset are credited above. Before redistributing this fine-tune, verify the applicable licenses and usage conditions for both the base model and the dataset, including any restrictions that apply to teacher-generated data.
The model is released under the Apache-2.0 license. However, the underlying dataset’s licensing terms and usage restrictions should be reviewed separately to determine whether they permit the intended use and redistribution of this fine-tune.
🙌 Acknowledgements
Thanks to the creators and maintainers of the Ministral base model and the Qwen distillation dataset for making this experiment possible.
Have fun experimenting, evaluating, and building on top of it! 🚀