Custom 57M Language Model

A custom 57.55M parameter causal language model with modern transformer architecture.

Model Details

• Parameters: 57,553,632 (57.55M)
• Architecture: 12-layer Transformer
• Hidden Size: 432
• Attention Heads: 8
• Head Dimension: 54
• Intermediate Size: 1,728
• Vocabulary Size: 50,257 (GPT-2 tokenizer)
• Max Sequence Length: 1,024

Architecture Features

• RoPE Positional Embeddings: Rotary Position Embedding (θ=10000.0)
• SwiGLU Activation: Swish-Gated Linear Unit in feed-forward networks
• RMSNorm: Root Mean Square Layer Normalization (ε=1e-06)
• Tied Embeddings: Input and output embeddings share weights
• Dropout: 0.1 dropout rate

Training Configuration

• Dummy Phase: 2 epochs, 1,000 samples, LR=0.0005
• C4 Phase: 3 epochs, 1,000 samples, LR=0.0003
• Optimizer: AdamW (weight_decay=0.1)
• Scheduler: Cosine Annealing
• Gradient Clipping: 1.0

Generation Parameters

• Temperature: 0.8
• Top-K: 50
• Top-P: 0.9
• Repetition Penalty: 1.1
• Max New Tokens: 100

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("your-username/custom-57m-language-model")
model = AutoModelForCausalLM.from_pretrained("your-username/custom-57m-language-model")

input_text = "The future of artificial intelligence"
inputs = tokenizer.encode(input_text, return_tensors='pt')
outputs = model.generate(
    inputs, 
    max_length=100, 
    temperature=0.8,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.1
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Training Dataset

• Primary: C4 (Colossal Clean Crawled Corpus)
• Warm-up: Synthetic dummy data for initial training

License

MIT License

Model Card

This model was trained as an educational demonstration of transformer architecture implementation with modern techniques like RoPE embeddings and SwiGLU activations.