language:
- en
license: apache-2.0
tags:
- text-generation
- math
- distilgpt2
- openwebmath
- arithmetic
datasets:
- openwebmath
metrics:
- exact_match
pipeline_tag: text-generation
DistilGPT2-Math
Model Description
deadMarkov/distilgpt2-math is a lightweight, causal language model specifically fine-tuned for mathematical reasoning and arithmetic. It is built upon the foundational DistilGPT2 architecture but includes structural modifications and targeted training to enhance its numerical and mathematical logic capabilities.
Key Modifications
- Modified Vocabulary: The tokenizer and model vocabulary were uniquely modified and optimized to focus heavily on mathematical symbols, notation, and reasoning-specific tokens.
- Supervised Fine-Tuning (SFT): The model was fine-tuned extensively on a mixture of OpenWebMath and highly targeted synthetic arithmetic data to reinforce step-by-step mathematical problem-solving.
Model Details
- Model Type: Causal Language Model
- Base Architecture: DistilGPT2 (~82M parameters)
- Language(s): English, Mathematics
- License: Apache 2.0
Evaluation Results
The model was evaluated using the EleutherAI Language Model Evaluation Harness (v0.4.11).
GSM8K (Grade School Math)
Performance on the GSM8K dataset using a 5-shot prompt setting (greedy decoding, temperature = 0.0):
Evaluation run details:
- Framework:
lm-evaluation-harness
- Model Precision:
torch.float32
- Device: CUDA (RTX A4000)
How to Get Started with the Model
You can use this model directly with the Hugging Face transformers library:
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "deadMarkov/distilgpt2-math"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "Question: If I have 5 apples and buy 7 more, how many do I have?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Uses & Limitations
Intended Uses
- Researching mathematical capability scaling in very small, distilled language models.
- Experimenting with specialized tokenization/vocabularies for arithmetic tasks.
- Serving as a lightweight baseline for small-scale math SFT experiments.
Limitations
- As an 82-parameter model, its mathematical reasoning capacity is heavily constrained compared to larger (7B+) models. It may struggle with complex, multi-step word problems or advanced calculus.
- The model may occasionally hallucinate numbers or lose track of arithmetic operations over long context windows.
