SCG-LM V2 (46M)

SCG-LM V2 is a 46,346,752-parameter Llama-style language model trained from random initialization on 45.0 billion tokens of human-written English. Training took 31.9 hours on a single H100. It is a base model: it continues text, and it has not been instruction-tuned or safety-tuned. It was built for Track 01 of the Global Innovation Build Challenge V2 (train a language model from scratch with at most 50M parameters).

Successor: SCG-LM V2.1 is this model with one short fine-tuning stage on the four benchmarks' training splits. It scores higher on them (four-task mean 52.76 vs 47.12), but its scores are task-tuned and its WikiText-103 perplexity is worse (25.46 vs 23.85). This repository is the pretraining-only model.

Model architecture

Parameters by component

Compute

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "davemaxuellkr/scglm-v2-46m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

prompt = "The water cycle begins when"
# Training documents are separated by <|eos|> (id 2); prefixing it matches training and evaluation.
ids = torch.tensor([[tok.eos_token_id] + tok.encode(prompt, add_special_tokens=False)])
out = model.generate(ids, max_new_tokens=60, do_sample=False, repetition_penalty=1.1,
                     pad_token_id=tok.pad_token_id, eos_token_id=tok.eos_token_id)
print(prompt + tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Tested with Transformers 4.46.3 and 5.5.4 (PyTorch 2.10). The special tokens are <|pad|> = 0, <|bos|> = 1, <|eos|> = 2 and <|unk|> = 3. To score text by likelihood, prepend <|eos|> as the first context token, the way the evaluation below does (prefix_token_id=2).

Evaluation

Zero-shot, raw accuracy (%) ± standard error, from EleutherAI lm-evaluation-harness v0.4.12 at a pinned commit. WikiText-103 perplexity is computed on the validation split with context 1,024 and stride 512 (269,569 scored tokens). The comparison row is our own earlier model (V1: same architecture, 13.0B training tokens) under the identical protocol.

Normalized accuracy (V2): HellaSwag 30.95, ARC-Easy 42.09, PIQA 60.66.

How to read these numbers:

• Perplexity: the WikiText-103 improvement over V1 is clear, about 6.5%.
• Accuracy: the 0.17-point gain in the four-task mean is within per-task standard errors, and PIQA is 1.3 points lower than V1.
• Near-chance tasks: at this scale HellaSwag and WinoGrande sit close to chance.

Selection and disclosure:

• The checkpoint was chosen on development data only (held-out text and held-out items from benchmark training splits) and frozen before any official V2 score was computed.
• A pre-registered rule then compared it with V1.
• V1's official scores had been observed earlier, and a second V2 variant (a different data mix in the final learning-rate decay) was evaluated and not selected (four-task mean 47.08, perplexity 23.56).

Training data

Human-written English only; no model-generated corpora.

Contamination controls.

• N-gram exclusion: any document sharing a 13-word n-gram with the train or evaluation splits of HellaSwag, ARC-Easy, PIQA or WinoGrande, or with the WikiText-103 validation and test splits, was removed.
• Domain blocks: all of wikihow.com and instructables.com (the sources of HellaSwag and PIQA) were blocked, removing 12,708 documents.
• Deduplication: exact, URL and MinHash near-duplicate removal across all sources.
• Limits: this is not a guarantee against paraphrased overlap.

Benchmark-targeted slice. A fastText classifier was trained to recognize text resembling benchmark training examples: 80% of the train splits of ARC-Easy/Challenge, PIQA, HellaSwag, WinoGrande, OpenBookQA, SciQ and CommonsenseQA, against length-matched web passages. The top 20% of pool documents by score form this slice. Benchmark items were never used as training text, and official evaluation splits were never used for selection or tuning.

Tokenizer. A 16,384-token byte-level BPE trained from scratch on 100M characters of FineWeb and English Wikipedia (training partitions only). A pre-registered comparison against new tokenizers trained on the V2 mixture kept this one.

Training procedure

Held-out loss plateaued near 3.17 nats during the constant phase, then fell to 2.97 during the final decay.

Limitations and risks

• Small model, frequent errors. It produces fluent but often false statements. For example, it continued "The water cycle begins when" with "…the water in the ocean is depleted". Do not use it for factual answers or advice.
• Not an assistant. There was no instruction tuning, preference tuning or safety tuning, so it does not follow instructions and has no content filtering.
• Data biases. Web, encyclopedic, Q&A and older public-domain book text carry the social biases and dated viewpoints of their sources.
• English only, with a 1,024-token context.
• Benchmark scope. Scores near chance on HellaSwag and WinoGrande mean these benchmarks say little about reasoning at this scale.

Intended uses: research and teaching on small-scale pretraining, data mixtures and training efficiency; a baseline for other from-scratch models; text-completion experiments.

Environmental impact

The selected run used 31.9 H100-hours. All V2 GPU work, including a 1B-token pilot and the unselected decay variant, came to 35.8 H100-hours. Power was not measured. At the H100 NVL's 400 W maximum board power, that is at most about 12.8 kWh for the selected run and 14.3 kWh in total. Carbon emissions depend on the local grid and were not measured.

Licenses and attribution

The model weights are released under the MIT license declared above. The training data keeps its own terms:

• FineWeb-Edu: ODC-By 1.0.
• DCLM-baseline: CC-BY-4.0.
• Wikipedia: CC-BY-SA 3.0 / GFDL.
• StackExchange content: CC-BY-SA 4.0.
• Project Gutenberg: public domain.
• Common Crawl terms also apply to the web sources.

We thank the maintainers of these datasets and of lm-evaluation-harness.

Files and reproducibility

• model.safetensors, config.json, generation_config.json, tokenizer.json: identical to the selected training export.
• tokenizer_config.json: identical except tokenizer_class, which is set to PreTrainedTokenizerFast so that Transformers 4.x can load the tokenizer. Token IDs are unchanged.
• figures/: the figures shown on this card, generated from the recorded results by src/scglm_v2/figures.py in the GitHub repository.
• training_provenance.json: the trainer's record of steps, token counts and data identity.
• eval_results.json: the official metrics above, with the SHA256 of the original evaluation record.
• Full records (code, configurations, per-phase registrations, the frozen development-only selection, and the evaluation outputs): https://github.com/davemaxuell/Global-Innovation-Build-Challenge-V2.

Citation

@misc{scglm_v2_2026,
  title  = {SCG-LM V2: a 46M-parameter language model trained from scratch},
  author = {davemaxuell},
  year   = {2026},
  url    = {https://huggingface.co/davemaxuellkr/scglm-v2-46m}
}

AI assistance: Claude (Anthropic) assisted the research, implementation, tests and documentation.