Qwen2.5-0.5B distilled on SST-2
Student: Qwen2.5-0.5B-Instruct. Teacher: Qwen2.5-7B-Instruct, used to write rationales.
Trained on two tasks: [label] (review -> sentiment) and [rationale] (review -> reasoning).
Results (SST-2 validation)
| Model | Params | Accuracy |
|---|---|---|
| Teacher: Qwen2.5-7B-Instruct (zero-shot) | 7B | 0.96 |
| Student before distillation (zero-shot) | 0.5B | 0.91 |
| Student after distillation (this model) | 0.5B | 0.945 |
The distilled student is about 14x smaller than the teacher and reaches nearly the same accuracy.
Usage
Prompt format for classification:
[label] Review: <text>
Sentiment:
Use [rationale] Review: <text>\nReasoning: to get an explanation instead.