Talon-D1-0.5B

A 480M parameter language model built from scratch. Every component - RMSNorm, RoPE, Grouped Query Attention, SwiGLU - was implemented manually without using the HuggingFace transformers library. The tokenizer is GPT-2's (not custom).

This isn't meant to compete with frontier models.

Model Variants

Architecture

480M Parameters
├── Token Embeddings (50257 × 1024)
├── 24 Transformer Blocks
│   ├── RMSNorm (pre-norm)
│   ├── Grouped Query Attention (16 heads, 8 KV heads)
│   │   └── RoPE positional embeddings
│   ├── SwiGLU FFN (1024 → 4096 → 1024)
│   └── Residual connections
├── Final RMSNorm
└── LM Head (1024 → 50257)

Training

Base Model

• Data: FineWeb-Edu (~10B tokens seen)
• Hardware: 8× A100 40GB (GCP)
• Framework: PyTorch + Accelerate (FSDP)
• Steps: 49,500
• Batch size: 262K tokens/step
• Learning rate: 3e-4 → 3e-5 (cosine decay)
• Final loss: 2.48

Instruction Fine-tuning

• Data: Stanford Alpaca (52K examples)
• Epochs: 13
• Learning rate: 2e-5 → 1e-5
• Final loss: 1.69

Benchmarks

Evaluated using lm-evaluation-harness:

Note: SmolLM2 trained on 4T tokens, Qwen2 on 7T tokens. Talon trained on 10B tokens (400-700× less data).

Usage

This model uses a custom architecture. You'll need the Talon codebase:

git clone https://github.com/SalahALHaismawi/Talon-v1
cd Talon-v1
pip install -r requirements.txt

Download weights

huggingface-cli download SalahALHaismawi/Talon-D1-0.5B --local-dir ./checkpoints

Chat (Instruct)

python scripts/chat.py --checkpoint checkpoints/instruct

Generate (Base)

python scripts/generate.py --checkpoint checkpoints/base --prompt "The key to learning is"

Instruction Format

### Instruction:
What is the capital of France?

### Response:

Limitations

• 480M parameters is small - the model will hallucinate facts confidently
• Alpaca fine-tuning - single-turn instructions only, no conversational ability
• No RLHF - doesn't know when to say "I don't know"

License

Apache 2.0