Model Card
Overview
This model is a 156M-parameter English-language causal language model trained on a large-scale text corpus and instruction-tuned for general question answering and task completion.
Running script
see github -> https://github.com/firdavsus/LLM_D2
Model Details
- Model size: 156M parameters
- Architecture: Transformer (causal LM)
- Tokenizer: GPT-2 tokenizer
- Languages: English only
Training Curves
Training Data
Pretraining
- Dataset: The Pile (10B token subset)
- Domain: mixed-domain text (web, books, articles, code, etc.)
Instruction Fine-tuning
- Dataset: Alpaca (cleaned subset)
- Size: ~50,000 instruction–response examples
- Formatting: instruction-style prompt/response pairs
Training Setup
Pretraining
- Steps: 218,000
- Final training loss: 2.6
Post-training (Instruction Fine-tuning)
- Steps: 2,500
- Final training loss: 1.9
Evaluation
| Benchmark | Score |
|---|---|
| HellaSwag | 28.5 |
Intended Use
- Instruction-style prompting
- Basic question answering
- Text generation and summarization
- Lightweight assistant-style tasks (English)
Limitations
- Small model size limits reasoning and factual reliability
- May produce incorrect or inconsistent answers
- Instruction-following quality depends strongly on prompt format
- Not suitable for high-stakes or safety-critical use
This model has not been safety-aligned. Please apply your own moderation and guardrails when deploying it ;)
FOR ADDITIONAL INFO CHEKC INFO.TXT