BananaMind-IDK-3M

A 2,715,984-parameter small language model trained from scratch on 8B tokens of FineWeb-Edu, requested by @Banaxi-Tech and trained by @Compactbot.

Architecture

Training

• Data: 8B tokens of FineWeb-Edu (cycled from a 3.6 GB local sample)
• Optimizer: AdamW, lr 5e-3, cosine decay with 200-step warmup
• Batch: 4 × grad-accum 8 = effective 32, seq 2048
• Steps: 122,070 (reached the 8B-token target)
• Hardware: RTX 5090 (shared, batch reduced to fit ~2 GB free VRAM)

Evaluation (zero-shot, loglikelihood)

These are honest numbers for a 2.7M-param model. For reference, chance is 50%/25%/25%/25% respectively.

Known limitations

This model produces degenerate greedy generation — outputs collapse into repetition loops after the first sentence. It learned token-level statistics (grammatical first sentences) but not enough structure to sustain coherent multi-sentence generation. This is expected at 2.7M params even with 8B tokens of training data.

The model is published as a research artifact / data point, not as a usable generation model.

Files

• final.pt — full checkpoint (state_dict + optimizer + step), load with torch.load('final.pt', map_location='cpu')['model']
• config.json — architecture config
• tokenizer.json — BPE tokenizer (8192 vocab, 5922 merges)
• tokenizer_config.json — tokenizer settings

Usage

import torch

ckpt = torch.load('final.pt', map_location='cpu')
state_dict = ckpt['model']  # 128 tensors, 2,715,984 learnable params

The model uses a custom architecture (GQA attention + SwiGLU FFN + RoPE + RMSNorm). A loading script is not included; the state_dict keys follow the pattern:

• model.embed_tokens.weight [8192, 144]
• model.layers.{i}.{attn|mlp|norm}.* for i in 0..5
• model.norm.weight [144]
• (lm_head is tied to embed_tokens)