✴️ K2-Horizon 3.7B — Claude Opus Reasoning Distilled : GGUF

Static quants in GGUF format of xtools-at/K2-Horizon-3.7B-Claude-Opus-Reasoning-Distilled

---

✴️ K2-Horizon 3.7B — Claude Opus Reasoning Distilled

A reasoning-focused fine-tune of IFM/K2-Horizon-3.7B, trained on 8,000 examples of Claude Opus and Sonnet reasoning traces.

K2-Horizon-3.7B is part of the Institute of Foundation Models' open-weight K2-Horizon family. It's a dense, decoder-only language model with a 3.7B core and approximately 5.06B stored parameters including embeddings. The original model supports a context window of up to 512K tokens and was developed with an emphasis on open research, reasoning, coding, and agentic capabilities. Its training resources and evaluation materials are also publicly available.

This fine-tune explores a simple question: can a compact, general-purpose model learn useful problem-solving patterns from high-quality reasoning traces without requiring an extensive additional training run?

The approach takes inspiration from TeichAI's Qwen3.5 Opus Reasoning Distill, with additional ideas from Jackrong's Qwen3.5 Opus Reasoning Distilled v2. Fine-tuning uses LoRA for one epoch over a mixed reasoning dataset, keeping the experiment relatively lightweight.

✨ What's the idea?

Producing a convincing explanation and actually solving a problem are two different things. This experiment aims to encourage the latter by exposing the model to examples of structured, multi-step reasoning.

The fine-tune focuses on patterns that may be useful for:

• 🧩 Multi-step problem-solving and logical reasoning.
• 💻 Coding and technical questions.
• 🔢 Mathematical and analytical tasks.
• 📝 Breaking down complex questions and working toward a coherent answer.

The goal isn't simply to make responses longer. It's to explore whether the model can learn more useful ways of approaching difficult problems.

These are intended use cases, not claims of measured improvement. Independent evaluations against the original K2-Horizon checkpoint are still needed to establish the effect of fine-tuning.

📚 Training data

The training corpus combines four datasets containing Claude-generated reasoning examples:

A total of 8,000 examples were used for training, alongside a separate 400-example evaluation set (5%) sampled proportionally to the dataset composition. All entries were filtered to a maximum length of 8,000 characters.

Mixing examples from different Claude model generations gives the training set some variety in reasoning styles rather than relying on a single source. As with any synthetic-data distillation, however, the model may also inherit recurring errors, stylistic biases, or weaknesses from its source traces.

🔬 Training setup

The model was fine-tuned using LoRA, targeting all LoRA-compatible model modules. Training was limited to a single epoch.

The single-epoch approach keeps the additional training relatively short and limits repeated exposure to the synthetic traces. This is a practical starting point for reasoning distillation, although it doesn't guarantee that the base model's existing capabilities will be preserved.

Note that the fine-tuning context length is 8,192 tokens. While the original K2-Horizon-3.7B supports a much longer context window, the full 512K-token capability has not been independently verified for this fine-tuned checkpoint.

📊 Training results

The evaluation loss ended up lower than the training loss. These values are useful as training diagnostics, but they aren't direct measures of reasoning quality and don't, by themselves, establish an improvement over the original model.

Benchmark comparisons and qualitative evaluations on unseen problems would help determine whether the fine-tune improves reasoning, coding, or mathematical performance.

🎯 Intended use

This model is intended for experimentation with compact reasoning models, including:

• Multi-step reasoning and analytical assistance.
• Coding and mathematical problem-solving.
• Local inference and resource-conscious deployments.
• Research into synthetic-data distillation and reasoning fine-tuning.

It can also serve as a starting point for further experiments with dataset composition, reasoning behavior, and fine-tuning strategies.

⚠️ Limitations

A few things to keep in mind:

• Reasoning isn't guaranteed to be correct. Detailed explanations can still contain mistakes, unsupported assumptions, or invalid conclusions.
• Synthetic-data bias. The model may reproduce stylistic habits, blind spots, and errors present in its source traces.
• General capabilities may shift. Fine-tuning can affect instruction-following, factual recall, and performance on tasks outside the training distribution.
• Evaluation is limited. The reported losses alone don't establish gains over the base model. No independent benchmark results are reported for this fine-tune.
• Length filtering. Filtering examples by character count may exclude useful long-form traces. An 8,000-character limit is not equivalent to an 8,192-token limit.
• Long-context behavior is unverified. The base model supports up to 512K tokens, but this experiment used an 8,192-token training context. Performance at the base model's maximum context length should be tested separately.

This is an experimental fine-tune rather than a replacement for comprehensive evaluation. For production use, test it against the original K2-Horizon model on representative tasks and check outputs carefully.

🙏 Acknowledgements

Thanks to the Institute of Foundation Models for releasing K2-Horizon and its supporting research resources, the Unsloth team for making efficient fine-tuning more accessible, and the authors of the TeichAI and Jackrong reasoning-distillation projects for sharing useful approaches and inspiration.

The dataset authors are credited above. Please review the underlying datasets' licenses and terms before redistributing this model.

---

The interesting question isn't whether a model can produce more reasoning text, but whether that reasoning actually helps it solve problems better. This fine-tune is one small experiment in exploring that question. 🧪