This is a decensored version of xtools-at/Qwen3.5-4B-MultiDistill-3xMerge, made using Heretic v2.0.0.dev0

[!TIP] This model is reproducible!

See the README in the reproduce directory for more information.

Arbitrary-Rank Ablation (ARA) parameters

Performance

---

Qwen3.5-4B Triple-Distill Reasoning Merge

A weighted merge of three reasoning-oriented Qwen3.5-4B fine-tunes, combining Claude Opus 4.6-style reasoning distillation, Empero's Qwen3.8 teacher distillation, and DeepSeek-V4-Pro mathematics and STEM reasoning.

The goal is to combine complementary training approaches in a compact model:

• General reasoning: Mathematics, logical deduction, and structured problem-solving.
• Reasoning efficiency: Concise, organized reasoning patterns distilled from Claude Opus 4.6-style traces.
• Broad instruction following: Teacher-generated reasoning traces covering general reasoning and instruction-following tasks.
• Mathematics and STEM: Detailed derivations, scientific problem-solving, and multi-step quantitative reasoning.

This is an experimental checkpoint merge. The combined model has not been independently benchmarked in this model card, and its capabilities should not be assumed to match the reported performance of any individual source model.

Model overview

Merge recipe

The merge combines three fine-tuned checkpoints using the following coefficients.

Merge implementation: Linear weighted average.

Source models

1. Claude Opus 4.6 reasoning distillation

Source model card

This fine-tune targets general-purpose reasoning with an emphasis on structured, economical reasoning traces.

Its author describes the v2 training run as using more than 14,000 Claude Opus 4.6-style reasoning samples. The stated objective is to improve reasoning structure, reduce redundant intermediate steps, and support transfer across mathematics, logic, general knowledge, and other analytical tasks.

The source model is fine-tuned from Qwen3.5-4B and uses curated reasoning-distillation data.

Role in this merge: Contributes general reasoning patterns and a focus on efficient, structured problem-solving.

2. Empero Qwen3.8-4B Distill

Source model card

Empero describes this model as a full-parameter distillation of its Qwen3.8 2.4T A95B teacher into the Qwen3.5-4B architecture.

The source card reports approximately 45,000 curated teacher traces covering mathematics, general reasoning, and instruction following. These traces are described as quality-filtered and intended to transfer teacher-generated reasoning behavior into a smaller model.

The source card also describes learned <think> blocks and native function-calling behavior following the Qwen3.5 specification.

Role in this merge: Adds a second teacher-distillation approach, broadening the training emphasis beyond the Claude-derived reasoning data.

3. DeepSeek-V4-Pro-Qwen3.5-4B

Source model card

This model is a Qwen3.5-4B fine-tune distilled from DeepSeek-V4-Pro responses generated in Max Effect mode.

Its author reports a training mixture of approximately 250,000 mathematics and STEM samples, emphasizing detailed derivations, decomposition of difficult problems, and construction of verifiable final answers.

The source card describes a training workflow using Unsloth-assisted fine-tuning and a LoRA-based training pipeline, with the resulting weights merged into BF16.

The author explicitly notes that the 4B SFT mixture did not include coding-specific training data. Coding and tool-use transfer are therefore not established by the source model's training description.

Role in this merge: Adds a concentrated mathematics and STEM reasoning component.

Base model

Qwen3.5-4B

Official model card

Qwen3.5-4B provides the shared underlying architecture for this merge.

The Qwen3.5 family supports language and visual understanding, reasoning, coding, and agent-oriented workflows. Its architecture combines gated attention with Gated Delta Networks, and the base model documents a native context length of 262,144 tokens.

These specifications describe the base model. They do not guarantee that every capability, context setting, or inference optimization will be preserved by this particular merge.

Training data and provenance

This checkpoint is produced by merging three independently fine-tuned models. It was not directly trained on a newly combined dataset.

Claude Opus 4.6 distillation datasets

The source model card identifies the following public datasets:

These are the publicly identified datasets associated with the Claude reasoning source. They are not necessarily exhaustive records of every preprocessing or training stage.

Empero Qwen3.8 distillation data

Empero reports approximately 45,000 curated teacher traces generated from its Qwen3.8 2.4T A95B teacher.

The documented subject areas include:

• Mathematics
• General reasoning
• Instruction following

The source card describes these as internal distillation datasets. Individual public dataset repository IDs, sample-level provenance, and detailed dataset composition are not specified in the card.

DeepSeek-V4-Pro distillation data

The DeepSeek-derived source model reports approximately 250,000 mathematics and STEM training samples generated from DeepSeek-V4-Pro in Max Effect mode.

The documented training focus includes:

• Mathematical derivations
• Scientific and quantitative reasoning
• Multi-step problem decomposition
• Verification and final-answer construction

The source card does not provide a complete list of publicly accessible dataset repository IDs for this training mixture.

Provenance caveat

The dataset information above describes the source models' reported training data. It does not imply that all three datasets or training mixtures were directly combined or that each source model used the same data.

For reproducibility, consult the source model cards and dataset repositories. Dataset licenses and usage restrictions should be checked independently before redistribution or commercial deployment.

Intended use

This merge is intended for experimentation with a compact reasoning model combining multiple teacher-distillation approaches.

Potential use cases include:

• Mathematical problem-solving: Arithmetic, algebra, word problems, and multi-step quantitative reasoning.
• STEM reasoning: Scientific questions and structured analytical problems.
• Logical deduction: Breaking complex questions into manageable steps and checking intermediate conclusions.
• General instruction following: Responding to questions and following structured task instructions.
• Analytical assistance: Comparing approaches, explaining solutions, and organizing technical information.

These are intended use cases based on the source-model training descriptions, not independently established performance guarantees for the merged checkpoint.

Reasoning behavior

The source models emphasize explicit reasoning traces, structured intermediate steps, and final-answer construction.

The merged model may produce visible reasoning markers such as <think> depending on its tokenizer, chat template, and inference configuration. The presence of such markers does not guarantee that the reasoning is correct or complete.

For reliable evaluation, assess both the final answer and its consistency with the original question. Generated explanations should not be treated as proof that a conclusion is correct.

Limitations

• Merge interference: Independently fine-tuned checkpoints may contain conflicting parameter updates. The merge may weaken capabilities that were present in one or more source models.
• Unverified benchmark performance: No independent benchmark results are reported for this merged checkpoint.
• Reasoning reliability: The model can produce plausible but incorrect derivations, unsupported assumptions, or inconsistent conclusions.
• STEM limitations: Strong source-model results on selected mathematical or scientific evaluations do not guarantee equivalent results on this merge or on broader scientific tasks.
• Coding capabilities: The DeepSeek-derived source's documented training mixture contains no coding-specific SFT data. General coding ability may still be present through the base model or other sources, but it has not been established for the merge.
• Tool use: Native function-calling support in a source model does not guarantee correct tool selection, argument formatting, or execution in the merged checkpoint.
• Multimodal compatibility: The base architecture supports image inputs, but the merge's visual capabilities depend on whether the necessary weights, processor, and configuration are retained correctly.
• Context length: The base model's native context limit is not a guarantee of the usable context length of this merge in every inference engine.
• Training-data limitations: Dataset quality, coverage, filtering, and possible contamination can affect generalization.
• High-stakes applications: The model has not been independently validated for medical, legal, financial, or other safety-critical uses.

Always independently verify important factual, mathematical, and scientific conclusions.

Evaluation

No independent evaluation results are reported for this merged checkpoint.

The source models have their own reported evaluations, but their scores should not be copied into this card as results for the merge.

A useful evaluation plan would compare the merged model against all three source models and the original Qwen3.5-4B checkpoint under consistent inference settings.

Suggested evaluation areas:

Record model revisions, quantization, prompts, inference parameters, evaluation datasets, and scoring methodology alongside any published results.

Acknowledgements

This merge builds on the work of:

• The Qwen team for the Qwen3.5 model family.
• Jackrong for the Claude Opus reasoning and DeepSeek-V4-Pro distillation models.
• Empero for the Qwen3.8-4B distillation model.
• The authors and maintainers of the underlying training datasets.
• The open-source fine-tuning and inference communities.

Please consult the original model cards for detailed acknowledgements and citations.

License

This repository declares the Apache-2.0 license.

Before redistributing the merged weights or using them commercially, verify the applicable license and usage terms for the base model, all source checkpoints, and the underlying training datasets.