MiniMind-MoE-198M-A64M-Pretrain-v1.0

Original final 2-epoch pretraining checkpoint, trained from random initialization on 2026-10-24 and recovered from the Runpod Global Volume on 2026-11-13. This is the exact base checkpoint used to train caspar/MiniMind-MoE-198M-A64M-Mixed-SFT-v1.0. Its original checkpoint SHA256 matches the mixed-SFT run's recorded base_sha256.

Architecture and training

MiniMind MoE: 198,416,640 total parameters, 4 experts with top-1 routing (~64M active), hidden size 768, 8 layers, vocabulary 6400, tied input/output embeddings. Architecture code is derived from jingyaogong/minimind under Apache-2.0; this release includes the original training-time model implementation and tokenizer.

FineWeb-Edu sample/10BT, revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, filtered to score >= 3.5 (1,288,228 documents before splitting). Each complete document is tokenized as BOS + text + EOS. Training uses complete 340-token windows with stride 339, drops each document's final short window, and never packs across documents. Training/validation split uses a hash of the full document text. Training has 4,407,066 windows per epoch. BF16 AdamW on RTX 5090, batch 64 x accumulation 4, 2 epochs, 34,432 optimizer updates, peak learning rate 3e-4. See training_summary.json.

Recorded validation PPL: 7491.43 before training -> 13.17 after training, on 1,024 fixed held-out windows (347,136 next-token targets). These are historical training-run metrics; publication checks reloaded weights and verified packaging, without rerunning the full validation dataset. This is an experimental text continuation model before SFT. Chat, instruction following, reasoning accuracy and contexts longer than the trained length of 340 are unvalidated. The tokenizer includes a chat template inherited from the tokenizer assets; its presence does not imply chat training. Source dataset licenses and terms continue to apply; the model/code license does not relicense the dataset.

Files and usage

• pretrain_768_moe.pth: byte-for-byte original FP16 PyTorch state dictionary (no optimizer state).
• model.safetensors, config.json, model_minimind.py: Transformers package of the same learned weights.
• Tokenizer, license, evaluation and training summaries, and release.json hashes.

Review model_minimind.py before enabling custom code. Install requirements.txt.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "caspar/MiniMind-MoE-198M-A64M-Pretrain-v1.0"
tokenizer = AutoTokenizer.from_pretrained(repo, revision="v1.0")
model = AutoModelForCausalLM.from_pretrained(
    repo, revision="v1.0", trust_remote_code=True, torch_dtype=torch.float32,
).eval()
x = tokenizer("The purpose of education is", return_tensors="pt", add_special_tokens=False).input_ids
with torch.inference_mode():
    y = model.generate(inputs=x, max_new_tokens=min(64, 340-x.shape[1]),
                       do_sample=False, eos_token_id=tokenizer.eos_token_id, pad_token_id=0)
print(tokenizer.decode(y[0], skip_special_tokens=True))

For native loading, use the bundled model code and torch.load("pretrain_768_moe.pth", map_location="cpu", weights_only=True) with load_state_dict(..., strict=True). No training data, optimizer states, credentials or W&B logs are included.