littlerock-1M
A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning.
Architecturally a stock Llama, so it loads with plain transformers — no
trust_remote_code, no custom modelling file:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/littlerock-1M")
model = AutoModelForCausalLM.from_pretrained("opencerebral/littlerock-1M")
Results
lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via
AxiomicLabs' bencharithmark-3.py (float32, default acc_norm).
| Benchmark | Score |
|---|---|
| HellaSwag | 26.43 |
| ARC-Easy | 30.98 |
| ARC-Challenge | 21.76 |
| PIQA | 52.23 |
| ArithMark-3 | 25.20 |
| Avg | 32.56 |
| Intelligence Index | 2.29 |
ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic.
A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all.
Architecture
| Parameters | 1,002,720 |
| Non-embedding | 609,504 |
| Layers | 6 |
| Hidden size | 96 |
| Heads | 6 (head dim 16) |
| FFN hidden (SwiGLU) | 224 |
| Context | 512 |
| Vocab | 4,096 |
| Position encoding | RoPE |
| Norm | RMSNorm |
| Embeddings | tied |
The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer's embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level BPE with byte fallback (no UNK is possible), fitted on 400k documents sampled from this model's own pretraining mixture.
Training
| Tokens | 5,000,000,000 |
| Mixture | 40% FineWeb-Edu, 60% DCLM-baseline |
| Steps | 19,073 |
| Batch | 262,144 tokens/step |
| Optimizer | AdamW (0.9, 0.95), wd 0.1 |
| LR | 3e-3 cosine → 3e-4, 50M-token warmup |
| Precision | bf16 autocast, fp32 master weights |
| Final val loss | 3.6251 (ppl 37.6) |
| Hardware | 1x RTX 3060, 1h 49m |
Data was materialised to a fixed local binary before training (no streaming), with documents EOS-separated, and the token budget enforced exactly at 5,000,000,000.
Limitations
This is a ~1M-parameter base model. It is a next-token predictor with no instruction tuning and no alignment, it has very limited world knowledge, and it will produce fluent but frequently false text. It exists to measure what a 1M-parameter budget can do.
Sample (prompt in italics): The capital of France is a national world of fashion, a country and many other nationwide nationality. In this case, the public has been banned in the early 17th century…
Copyright & License
Copyright 2026 OpenCerebral
This project and all associated files (the "Work") are licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at:
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.