kaggle-llm-124m-base
A 124M-parameter base language model pretrained from scratch on 1.92B tokens of FineWeb-Edu using free Kaggle GPUs (2x NVIDIA T4, about 21 hours over two 12-hour sessions). Validation loss 3.30 (perplexity 27).
This is a text-completion model, not a chat model. For the instruction-following version see somdevsheel/kaggle-llm-124m-chat.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "somdevsheel/kaggle-llm-124m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16).to("cuda")
ids = tok("Photosynthesis is the process", return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=100,
do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))
Model
| Architecture | Decoder-only Transformer, LLaMA-style: RoPE, RMSNorm (pre-norm), SwiGLU, no biases, tied embeddings |
| Size | 123.6M parameters (85.0M non-embedding) |
| Shape | 12 layers, hidden 768, 12 heads, FFN 2048, context 1024 |
| Tokenizer | GPT-2 BPE (vocab padded to 50,304) |
Training
- Data: 1.92B tokens of FineWeb-Edu (sample-10BT), no token seen twice.
- Two runs of 524,288 tokens per step; PyTorch DDP on 2x T4; fp16 mixed precision with loss scaling.
- Run 1: 1,865 steps from scratch, peak LR 1.2e-3, 200-step warmup, cosine decay to 10% (val loss 3.43).
- Run 2 (continued pretraining): 1,796 more steps on unseen data, starting from run 1's weights, peak LR 6e-4 with a fresh cosine schedule.
- AdamW (0.9, 0.95), weight decay 0.1. Each schedule was fitted to the 12-hour Kaggle session limit by timing the first steps.
| Step | 0 | 250 | 500 | 1000 | 1500 | 1864 | 2865 | 3365 | 3661 |
|---|---|---|---|---|---|---|---|---|---|
| Val loss | 10.97 | 4.88 | 4.05 | 3.65 | 3.48 | 3.43 | 3.38 | 3.32 | 3.30 |
Steps after 1864 are the continued-pretraining run; its loss briefly rises (3.48 at step 2115) when the higher learning rate restarts, then falls below run 1.
Limitations
Trained on about 16 tokens per parameter, still short of compute-optimal (~20), so it has limited world knowledge and frequently produces fluent but incorrect statements. English only; no safety filtering beyond FineWeb-Edu's educational filter.