A 124M-parameter base language model pretrained from scratch on 1.92B tokens of FineWeb-Edu using free Kaggle GPUs (2x NVIDIA T4, about 21 hours over two 12-hour sessions). Validation loss 3.30 (perplexity 27).
This is a text-completion model, not a chat model. For the instruction-following version see somdevsheel/kaggle-llm-124m-chat.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "somdevsheel/kaggle-llm-124m-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16).to("cuda")
ids = tok("Photosynthesis is the process", return_tensors="pt").input_ids.to("cuda")
out = model.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=100,
do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))
| Architecture | Decoder-only Transformer, LLaMA-style: RoPE, RMSNorm (pre-norm), SwiGLU, no biases, tied embeddings |
| Size | 123.6M parameters (85.0M non-embedding) |
| Shape | 12 layers, hidden 768, 12 heads, FFN 2048, context 1024 |
| Tokenizer | GPT-2 BPE (vocab padded to 50,304) |
| Step | 0 | 250 | 500 | 1000 | 1500 | 1864 | 2865 | 3365 | 3661 |
|---|---|---|---|---|---|---|---|---|---|
| Val loss | 10.97 | 4.88 | 4.05 | 3.65 | 3.48 | 3.43 | 3.38 | 3.32 | 3.30 |
Steps after 1864 are the continued-pretraining run; its loss briefly rises (3.48 at step 2115) when the higher learning rate restarts, then falls below run 1.
Trained on about 16 tokens per parameter, still short of compute-optimal (~20), so it has limited world knowledge and frequently produces fluent but incorrect statements. English only; no safety filtering beyond FineWeb-Edu's educational filter.