🧠 c4-gpt-mini-causallm

A 44.91M-parameter GPT-2 Causal Language Model (BF16 + SDPA), designed, tokenized, and trained entirely from scratch on C4.

A 44.91M-parameter autoregressive causal language model built and trained 100% from scratch (zero pretrained weights) over 5 Epochs (6,220 steps — 203.74M dataset tokens) on 1× NVIDIA A100-SXM4 (40GB) using Native BF16 + TF32 + PyTorch SDPA, achieving Validation Perplexity 93.52 (Loss 4.5382) and Held-Out Test Perplexity 96.97 (Loss 4.5744).

---

✨ Highlights

• 🔨 100% From Scratch: No pretrained checkpoint initialization; all 44,909,056 parameters and the 50,257-token Byte-Level BPE tokenizer (fit strictly on Train split) are learned from scratch.
• 🎯 Sub-94 Validation & Sub-97 Held-Out Test PPL: Reduced Perplexity by 99.82% vs random initialization (54,773.00 Test PPL), with a small Val-to-Test generalization gap of +0.0362 nats (93.52 Val / 96.97 Test PPL).
• ⚡ Ampere Native BF16 + SDPA Acceleration: Reached 124,127 tokens/sec (242.44 blocks/sec) at 14.07 GB Peak VRAM, completing all 5 epochs in 27.37 minutes of training (30.37 minutes full end-to-end pipeline).
• 🪶 Compact 85.66 MB BF16 Footprint: Serialized natively in bfloat16 (model.safetensors: 89,825,728 bytes), well-suited for edge inference or Draft Model experiments in Speculative Decoding.
• 📚 Boundary-Aware Zero-Padding Packing: Strict non-overlapping 90,000 Train / 5,000 Val / 5,000 Held-Out Test split (seed=42) with explicit </s> (eos_token_id=2) document delimiters and linear-time packing (99.94% token retention, 0% <pad>).
• 🔓 Permissive Open Source: Released under the MIT License.

---

🚀 Quick Start

Generate text using PyTorch and 🤗 Transformers (stored natively in bfloat16):

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "GiangSon-5/c4-gpt-mini-causallm"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device).eval()

prompt = "Artificial intelligence is transforming"
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        top_p=0.92,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Or run via the high-level Hugging Face Pipeline:

from transformers import pipeline

generator = pipeline("text-generation", model="GiangSon-5/c4-gpt-mini-causallm")
result = generator("The future of software engineering lies in", max_new_tokens=60, do_sample=True, temperature=0.8, top_k=50, top_p=0.92)
print(result[0]["generated_text"])

---

💬 Sample Generations

Verbatim continuations generated after reloading the standalone bfloat16 checkpoint from disk (max_new_tokens=80, temperature=0.8, top_k=50, top_p=0.92, repetition_penalty=1.15):

Qualitative Critique: The model produces fluent English syntax and demonstrates clean autonomous EOS termination (</s> stop observed naturally in Prompt 2). Consistent with a compact 44.91M-parameter model trained on C4 web crawls, longer completions exhibit topical drift (Prompt 1 shifting into film promotional text with C4 anonymized 0000 year placeholders) and occasional local phrase repetition (to be able to be able, concepts, concepts).

---

1. Architecture Specs

---

2. Training Data & Procedure

Data Engineering

• Dataset: datablations/c4-filter-small (100,000 quality-filtered English web documents from the Colossal Clean Crawled Corpus — C4).
• Non-Overlapping 3-Way Split (seed=42):
  • Train: 90,000 documents → 79,585 packed 512-token blocks (40,747,520 tokens/epoch; 99.94% retention from 40,773,413 raw tokenized+EOS tokens).
  • Validation: 5,000 documents → 4,623 packed 512-token blocks (2,366,976 tokens).
  • Held-Out Test: 5,000 documents → 4,535 packed 512-token blocks (2,321,920 tokens).
• Total Tokens Trained Across 5 Epochs: 203,737,600 dataset tokens (79,585 × 512 × 5; 203,816,960 batch step-slot tokens across 6,220 steps × 64 blocks × 512 tokens).
• Preprocessing Strategy: Boundary-Aware Zero-Padding Sequence Packing. Each document ends with explicit </s> (eos_token_id=2) before linear concatenation (itertools.chain) into uniform 512-token chunks, completely eliminating <pad> tokens.

Training Configuration

---

3. Evaluation

---

4. Limitations & Ethical Considerations

• Model Capacity (~45M): Generates grammatically coherent English prose, yet remains constrained on multi-step factual reasoning, exhibits topical drift over longer generations, and shows occasional local phrase repetition.
• C4 Anonymization & Web Bias: Clean C4 crawl data contains numeric anonymization artifacts (e.g., 0000 replacing years or phone digits) and inherits societal biases from public web pages.
• Hallucination: Generates next-token continuations purely probabilistically without factual memory or external retrieval grounding.
• Intended Use: Base English (en) causal language model for research, educational benchmarking, and lightweight speculative decoding drafts; not aligned (no SFT/RLHF) and unsuitable for medical, legal, or financial advice.

---

5. Google Colab & Reproducibility

---

6. Citation & References

• 📄 C4 Dataset (Google Research): Colin Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (JMLR 2020). arXiv:1910.10683 | PDF Link
• 📄 Data-Constrained Scaling (Hugging Face / Harvard): Niklas Muennighoff et al., Scaling Data-Constrained Language Models (NeurIPS 2023). arXiv:2305.16264 | PDF Link
• 📄 GPT-2 Architecture (OpenAI): Alec Radford et al., Language Models are Unsupervised Multitask Learners (OpenAI Technical Report 2019). OpenAI PDF

---

7. License & Author

• License: Released under the MIT License.
• Author: Giang Sơn (GiangSon-5).