Tiny-GPT (51M Causal Language Model)
Tiny-GPT is a 51-million parameter causal language model built and trained from scratch on ~3.05 billion token positions across 4 pretraining stages using FineWeb and FineWeb-Edu. It includes a dual-mix Supervised Fine-Tuning (SFT) ablation study comparing conversational dialogue data (Mix A) against question-answering data (Mix B).
Project Objective & Findings: The primary goal was to train a capable conversational AI model from scratch. Evaluated against general assistant standards, the 51M architecture exhibits clear capacity limitations—hallucinating facts, struggling with symbolic math/logic, and failing Python syntax generation. However, the project successfully validates and documents the entire end-to-end small-LLM lifecycle: multi-GPU pretraining with gradient accumulation, deterministic validation caching, response-only SFT masking, isolated multi-config benchmarking, and local FP16 GGUF inference.
Architecture Specifications
| Specification | Value |
|---|---|
| Architecture | GPT-2 style decoder-only Transformer (GPT2LMHeadModel) |
| Parameters | 51,082,752 |
| Layers | 8 |
| Attention Heads | 8 |
| Hidden Dimension ($d_{\text{model}}$) | 512 |
| Context Length | 256 tokens |
| Tokenizer | GPT-2 BPE (vocab size: 50,257, EOS ID: 50,256) |
| Training Budget | 3,050,037,248 processed token positions (~59.71 tokens/param) |
| Final Pretraining Validation Loss | 3.6472 (Perplexity $\approx$ 38.37) |
| Export Format | FP16 GGUF (unquantized, ~99.5 MiB) |
Model Variants
- Mix A (
TinyGPT-Mix-A-F16.gguf): 70% Chat / 30% QA mixture (~1.20M active supervised tokens). Achieved a chat validation loss of 2.6181 (PPL 13.71). Demonstrates natural conversational turn-taking, empathetic acknowledgments, and polite hedging. - Mix B (
TinyGPT-Mix-B-F16.gguf): 30% Chat / 70% QA mixture (~0.59M active supervised tokens). Achieved a Q&A validation loss of 3.1494 (PPL 23.32). Geared toward direct question answering.
Running in LM Studio / llama.cpp
1. Download
Download either TinyGPT-Mix-A-F16.gguf (recommended) or TinyGPT-Mix-B-F16.gguf from the Files and versions tab and import it into LM Studio.
2. Prompt Template
Configure the prompt format to match the SFT training schema:
User: {prompt}
Assistant: