Qwen3-0.6B-Distill-Opus-v1
Overview
This is an experimental distillation of Qwen3-0.6B-Base. The model was trained using a LoRA adapter (r=24) on a single NVIDIA T4 GPU. The goal was to transfer encyclopedic and linguistic knowledge from Claude Opus 4.7 and ChatGPT 5.4 into a compact 596M parameter architecture. (it was fail i guess)
Technical Specifications
- Parameters: 596M
- Training Method: LoRA (r=24, alpha=48)
- Base Model: Qwen3-0.6B-Base
- Hardware: NVIDIA T4
Evaluation Results
The following table compares this distilled version against stock models and competitors in the sub-2B category.
The distill model was in Q4 quant!!! (approx 1-3% lie)
| Model | MMLU (5-shot) | GSM8K (Math) | Parameters |
|---|---|---|---|
| Qwen3-0.6B Distil | 44.57 | 6.00 | 596M |
| Qwen3-0.6B (Official) | 52.81 | 59.59 | 596M |
| Llama-3.2-1B (Official) | 39.00 | 44.40 | 1.2B |
| Gemma-2-2B (Official) | 51.30 | 61.10 | 2.6B |
| Qwen2.5-0.5B (Official) | 37.90 | 40.10 | 0.5B |
| SmolLM2-1.7B (Official) | 45.00 | 40.00 | 1.7B |
Detailed MMLU Breakdown (57 Categories)
Final Average Accuracy: 46.23%
| Category | Accuracy (%) |
|---|---|
| Marketing | 70.09 |
| Sociology | 63.18 |
| Computer Security | 63.00 |
| High School Psychology | 62.94 |
| High School Geography | 60.61 |
| High School Biology | 59.68 |
| International Law | 59.50 |
| Management | 59.22 |
| High School Microeconomics | 57.56 |
| High School European History | 56.97 |
| Security Studies | 55.10 |
| High School Government And Politics | 54.92 |
| Electrical Engineering | 54.48 |
| Clinical Knowledge | 54.34 |
| Medical Genetics | 54.00 |
| Us Foreign Policy | 54.00 |
| Nutrition | 53.27 |
| High School Computer Science | 52.00 |
| College Medicine | 50.29 |
| College Computer Science | 50.00 |
| Miscellaneous | 49.43 |
| Professional Medicine | 48.16 |
| Human Sexuality | 47.33 |
| High School World History | 47.26 |
| High School Macroeconomics | 46.67 |
| High School Us History | 46.57 |
| High School Chemistry | 46.31 |
| Jurisprudence | 46.30 |
| Astronomy | 46.05 |
| Public Relations | 45.45 |
| High School Statistics | 45.37 |
| Anatomy | 45.19 |
| College Biology | 45.14 |
| Moral Disputes | 45.09 |
| World Religions | 45.03 |
| Philosophy | 44.69 |
| Prehistory | 42.90 |
| Conceptual Physics | 42.55 |
| College Chemistry | 42.00 |
| Professional Psychology | 40.52 |
| Virology | 40.36 |
| Human Aging | 39.01 |
| Professional Accounting | 39.01 |
| Business Ethics | 39.00 |
| College Mathematics | 38.00 |
| High School Physics | 37.75 |
| Logical Fallacies | 36.81 |
| Formal Logic | 35.71 |
| Machine Learning | 35.71 |
| Elementary Mathematics | 35.19 |
| Econometrics | 33.33 |
| Abstract Algebra | 33.00 |
| Professional Law | 31.49 |
| High School Mathematics | 31.48 |
| College Physics | 28.43 |
| Moral Scenarios | 27.49 |
| Global Facts | 20.00 |
Knowledge Density (Perplexity)
A significant drop in Perplexity (PPL) indicates successful knowledge compression from the teacher models:
- Stock PPL: 10.3566
- Distilled PPL: 8.9864
- Delta: -1.3702
Limitations and Critical Warnings
- Hallucinations: The model is highly prone to severe factual errors in conversational contexts.
- Reasoning Decay: Mathematical logic (GSM8K) has significantly degraded compared to the stock base model.
- Inference Stability: Users may encounter token loops or "stuck" states. Thinking tags/Reasoning chains are not supported in this version.
Recommended Inference Settings
For stable results, use the following configuration:
- Temperature: 0.35
- System Prompt:
You - ai assistant - Stop Strings:
### Question:,### Response: - Repetition Penalty: 1.1 - 1.2