gpt2-124m-distilled-from-mistral7b
GPT-2 (124M) distilled from mistralai/Mistral-7B-Instruct-v0.2 with logit distillation.
Method
- Loss = 0.5 x T^2 x KL(teacher || student) + 0.5 x cross-entropy, temperature T = 2.0
- The tokenizers differ (GPT-2: 50,257 tokens, Mistral: 32,000), so the KL term is computed only over the 21,366 tokens whose text is identical in both vocabularies (renormalised), and only at positions where both tokenizers have a token boundary at the same character.
- Teacher loaded in 4-bit NF4. Data: Salesforce/wikitext/wikitext-2-raw-v1 (19,687 passages), 2000 optimizer steps.
Results (WikiText-2 test, cleaned text, 512-token windows)
| Perplexity | |
|---|---|
| GPT-2 before distillation | 28.98 |
| After distillation | 29.64 |
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("Sahilkumarr/gpt2-124m-distilled-from-mistral7b")
model = AutoModelForCausalLM.from_pretrained("Sahilkumarr/gpt2-124m-distilled-from-mistral7b")