✴️ Ministral 3 3B — Claude Opus Reasoning Distilled

A small reasoning-focused fine-tune of Ministral 3 3B, trained on 8,000 examples of Claude Opus and Sonnet reasoning traces.

The goal is straightforward: bring more structured, deliberate, multi-step reasoning to a compact model without throwing away the capabilities that make a small model useful in the first place.

This experiment builds on unsloth/Ministral-3-3B-Reasoning, taking inspiration from TeichAI's Qwen3.5 Opus Reasoning Distill and incorporating additional ideas from Jackrong's Qwen3.5 Opus Reasoning Distilled v2.

Rather than training for multiple epochs, this run uses a single epoch over a mixed reasoning dataset. The aim is to encourage useful reasoning patterns while keeping fine-tuning relatively conservative.

✨ What's the idea?

Small models can be surprisingly capable, but getting them to work through complex problems consistently is a different challenge from simply producing fluent answers.

This fine-tune explores whether high-quality reasoning traces can help a 3B-class model improve its approach to:

• 🧩 Multi-step problem-solving and logical reasoning.
• 💻 Coding and technical questions.
• 🔢 Mathematical and analytical tasks.
• 📝 Following a problem from initial interpretation through to a coherent answer.

The focus is on transferring useful problem-solving patterns, not simply making responses longer. Actual improvements across these tasks have not yet been established through independent benchmarking.

📚 Training data

The training corpus combines four datasets containing Claude-generated reasoning examples:

A total of 8,000 examples were used for training, with a separate 400-example evaluation set (5%) sampled proportionally to the dataset composition. All entries were filtered to a maximum length of 8,000 characters.

The mixture combines reasoning examples from different Claude model generations and datasets, providing some diversity in problem-solving styles rather than relying on a single source.

As with any synthetic-data distillation, the resulting model may inherit both the strengths and the weaknesses of its source traces.

🔬 Training setup

The model was fine-tuned using LoRA, targeting all LoRA-compatible model modules. The run was deliberately limited to one epoch.

The single-epoch approach follows a useful lesson from previous reasoning-distillation experiments: more training is not automatically better. A conservative run can help limit overfitting to synthetic traces, although it does not guarantee that the original model's capabilities will be preserved.

📊 Training results

The evaluation loss is lower than the training loss for this run. These values are useful as training diagnostics, but they should not be interpreted as direct measures of reasoning ability or as evidence of improved benchmark performance.

Independent evaluations would be needed to establish how well the fine-tune generalizes to unseen reasoning, coding, and mathematical tasks.

🎯 Intended use

This model is intended for experimentation with compact reasoning models, including:

• Multi-step reasoning and analytical assistance.
• Coding and mathematical problem-solving.
• Local inference and resource-conscious deployments.
• Research into synthetic-data distillation and reasoning fine-tuning.

It can also serve as a starting point for further experiments with dataset composition, reasoning length, and fine-tuning strategies.

Usage

Required base Mistral Reasoning system prompt (always set this):

# HOW YOU SHOULD THINK AND ANSWER

First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.

Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.

You can append additional instructions to the base system prompt.

⚠️ Limitations

A few things to keep in mind:

• Reasoning is not guaranteed to be correct. Detailed explanations can still contain mistakes, unsupported assumptions, or invalid conclusions.
• Synthetic-data bias. The model may reproduce stylistic habits, blind spots, and errors present in its source traces.
• General capabilities may shift. Fine-tuning can affect instruction-following, factual recall, and performance on tasks outside the training distribution.
• Evaluation is limited. The reported losses alone do not establish gains over the base model. No independent benchmark results are reported here.
• Length filtering. Filtering examples by character count may exclude useful long-form traces, and an 8,000-character limit is not equivalent to an 8,192-token limit.
• Multimodal capabilities are not specifically validated. Although the underlying Ministral model supports vision, this fine-tuning experiment focuses on reasoning text. Vision performance should be tested separately.

This is an experimental fine-tune rather than a replacement for comprehensive evaluation. For production use, test it against the base model on representative tasks and check outputs carefully.

🙏 Acknowledgements

Thanks to the Unsloth team for making efficient fine-tuning more accessible, and to the authors of the TeichAI and Jackrong reasoning-distillation projects for sharing useful approaches and inspiration.

The dataset authors are credited above. Please review the underlying datasets' licenses and terms before redistributing this model.

---

Built as a small experiment in transferring reasoning patterns to a compact model. The interesting question isn't whether it can produce more reasoning text, but whether that reasoning actually helps it solve problems better. 🧪