TinyTurk v2.7d FineWeb 40M: A 42M-Parameter English Decoder-Only Transformer

A 42,004,992-parameter English decoder-only Transformer language model trained from scratch on a subset of FineWeb-Edu. Reaches 73.66% on full BLiMP and performs below GPT-2 Small (124M) across all six benchmarks under a unified 0-shot lm-eval pipeline.

TinyTurk v2.7d FineWeb 40M is a compact causal language model. It is intended as a controlled small-scale language-modeling experiment examining the interaction between model capacity, training-corpus composition, and modern Transformer components (RoPE, RMSNorm, gated Micro-FFN, weight tying).

Model language. Despite the name "TinyTurk", this specific model is trained and evaluated in English only.

Model naming. The name "40M" denotes an approximate parameter size. The exact trainable parameter count is 42,004,992.

Evaluation methodology. All benchmark results reported below were produced with the EleutherAI Language Model Evaluation Harness (lm-eval) under a single unified pipeline. GPT-2 Small was re-evaluated with the same harness, same task versions, same prompts, same 0-shot setting, same batch size, and same random seed to enable a direct apples-to-apples comparison. cache_requests=False was set for all evaluations to prevent response caching. Benchmark numbers are not taken from external literature.

Model Description

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: English
• Training: From scratch (no pretrained initialization, no instruction tuning)
• Training corpus: 2,990,000-sample subset of FineWeb-Edu (3,000,000 drawn, 10,000 held out for validation)
• Model license: Apache 2.0
• Dataset license (FineWeb-Edu): ODC-By 1.0
• Exact trainable parameters: 42,004,992
• Rounded size designation: 40M

Headline comparison: TinyTurk 42M vs GPT-2 Small 124M

Both models were evaluated under the identical lm-eval pipeline (0-shot, cache_requests=False, same task versions, same batch size).

Note on the aggregate row. The final row is an unweighted mean of the reported metrics, not a standard composite benchmark score. Some tasks use acc and one (OpenBookQA) uses acc_norm, so this row is provided as a rough summary only and should not be treated as a primary performance figure.

Interpretation. TinyTurk 42M is consistently below GPT-2 Small across all six benchmarks, with an unweighted mean gap of -0.0406 accuracy points. The parameter-count difference (42M vs 124M) may contribute substantially to this gap, but this comparison does not isolate parameter count from other differences between the two models (training corpus, tokenizer, training procedure, and architectural details all differ). The smallest gap is on OpenBookQA (acc_norm, -0.0020) and the largest on BLiMP (acc, -0.0849).

Architecture

Training

Data

• Source dataset: FineWeb-Edu (HuggingFaceFW/fineweb-edu)
• FineWeb-Edu description: FineWeb-Edu is a filtered subset of FineWeb designed to retain web documents with higher educational quality scores. The full FineWeb-Edu corpus contains approximately 1.3 trillion tokens.
• Subset drawn: 3,000,000 text chunks
• Validation holdout: 10,000 chunks held out before training
• Actual training chunks: 2,990,000
• Nominal corpus token count (3M x 256): 768,000,000 tokens
• Actual training token count (2.99M x 256): 765,440,000 tokens
• Context length: 256 tokens per chunk
• Tokenizer training data: the training split of the 2.99M-sample subset only (validation chunks excluded from tokenizer training)
• Training epochs: 1

Important wording. The model was trained on a subset of FineWeb-Edu, not on the full FineWeb-Edu corpus. The tokenizer was trained on the training split only; validation and test chunks were not included in tokenizer training.

Optimization

Tokens-per-parameter ratio. 765.44M training tokens / 42.005M parameters = approximately 18.2 tokens per parameter.

Single-run disclosure. All reported training results are from a single training run with seed 42.

Validation

• Best validation loss: 3.1095
• Validation perplexity: 22.41 (exp(3.1095) = 22.41)
• Validation split size: 10,000 chunks held out from the same 3M subset before training

Evaluation

Methodology

All results were produced with the EleutherAI Language Model Evaluation Harness (lm-eval).

• Harness version: lm-eval==0.4.5
• Datasets library: datasets==3.1.0
• Transformers: transformers==4.46.0
• PyTorch: 2.14.0+cu126
• Device: NVIDIA GTX 1080 Ti (CUDA)
• Few-shot setting: 0-shot for all tasks
• Response cache: disabled (cache_requests=False)
• GPT-2 Small reference: evaluated with the identical harness, task versions, 0-shot setting, batch size, and seeds. Model ID: gpt2 (124M parameters).

Full result details

Standard errors are reported by lm-eval.

Primary BLiMP result: 0.7366 ± 0.0017 (full BLiMP, 67,000 minimal pairs, lm-eval 0.4.5, 0-shot).

Language Modeling

Validation perplexity is reported as an internal language-modeling diagnostic. It is not used for cross-model comparison.

Scientific Observations

The empirical observations of this release are:

1. TinyTurk 42M is consistently below GPT-2 Small across all six lm-eval benchmarks under an identical evaluation pipeline. The unweighted mean gap over the six reported metrics is -0.0406 accuracy points (0.4537 vs 0.4943).

2. The gap is smallest on OpenBookQA (acc_norm: -0.0020) and largest on BLiMP (acc: -0.0849).

3. The parameter-count difference (42M vs 124M) may contribute substantially to the observed gap, but this comparison does not isolate parameter count from other differences between the two models (training corpus, tokenizer, training procedure, and architectural details all differ).

4. No claim of superiority over GPT-2 Small is made. The comparison is provided as an apples-to-apples baseline under a single harness.

What This Model Does Not Claim

This release does not claim that:

• TinyTurk 42M matches or outperforms GPT-2 Small on any benchmark.
• The observed gaps are caused solely by parameter count or solely by corpus choice.
• Benchmark performance demonstrates factual reliability, reasoning ability, or general intelligence.
• The Micro-FFN component is novel or generally beneficial.
• FineWeb-Edu is generally superior to mixed-domain training.

Known Limitations

• Small model capacity. 42M parameters substantially limits factual recall, reasoning depth, and long-range generation quality.
• Short context. Maximum context length is 256 tokens.
• No instruction tuning. This is a pretrained causal language model, not an instruction-following assistant.
• English only. Trained and evaluated as an English language model.
• Factual reliability. Generated statements may be incorrect or fabricated.
• Single-seed results. All numbers are from one training run (seed 42).
• No Micro-FFN ablation. The effect of the Micro-FFN gate value is not measured in this release.
• Consistently below GPT-2 Small on all reported benchmarks. The gap is not closed by any single benchmark.

Usage

Important: The micro_gate value must be set to 0.75 to match the trained model. The default in TinyTurkV27Config is 0.0, which would disable the Micro-FFN entirely.

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_16k.model')

config = TinyTurkV27Config(
    vocab_size=16384,
    context_length=256,
    d_model=512,
    n_heads=8,
    n_layers=8,
    d_ff=2048,
    micro_gate=0.75,
)

model = TinyTurkGPTV27(config)
state = torch.load('pytorch_model.bin', map_location='cpu')
model.load_state_dict(state)
model.eval()

prompt = 'The history of the Roman Empire'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(200):
        logits, _ = model(x[:, -256:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]], dtype=torch.long)], dim=1)

print(tok.decode(x[0].tolist()))

Reproducibility

Provided

• Model configuration, exact parameter count, tokenizer size, context length.
• Training: dataset, subset size, validation holdout, actual training chunk count, nominal and actual token counts, batch size, LR, schedule, optimizer, weight decay, gradient clipping, epochs, seed, hardware, training time.
• Evaluation: harness version, task list, 0-shot setting, metric names, standard errors, cache_requests=False setting.
• Validation loss (3.1095) and perplexity (22.41).

Not provided in this release

• Exact FineWeb-Edu dataset revision (commit hash).
• Dataset sampling method.
• Tokenizer training hyperparameters.
• Multi-seed variance.
• Micro-FFN ablation.
• Raw per-example predictions.
• Baseline (GPT-2 Small) raw results and exact command.

Files

License

• Model weights and code: Apache License 2.0.
• Training data: FineWeb-Edu, released under ODC-By 1.0.

Citation

@misc{tinyturk-v27d-fineweb-40m,
  title  = {TinyTurk v2.7d FineWeb 40M: A 42M-Parameter English Decoder-Only Transformer},
  author = {stunmuffin},
  year   = {2026},
  note   = {Trained from scratch on a 2.99-million-sample training split of FineWeb-Edu}
}

References

• Gao, L., Tow, J., Abbasi, B., et al. (2024). The Language Model Evaluation Harness. Zenodo.
• Warstadt, A., Singh, A., & Bowman, S. R. (2020). BLiMP. TACL.
• Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). HellaSwag. ACL.
• Clark, P., Cowhey, I., Etzioni, O., et al. (2018). Think You Have Solved Question Answering? Try ARC. arXiv:1803.05457.
• Bisk, Y., Zellers, R., Gao, J., et al. (2020). PIQA. AAAI.
• Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018). OpenBookQA. EMNLP.
• Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021). WinoGrande. AAAI.
• Penedo, G., Kydlicek, H., Ben Allal, L., et al. (2024). The FineWeb Datasets. arXiv:2406.17557.
• Zhang, B., & Sennrich, R. (2019). RMSNorm. NeurIPS.
• Su, J., Lu, Y., Pan, S., et al. (2021). RoFormer. arXiv:2104.09864.
• Radford, A., Wu, J., Child, R., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
• Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR.
• Kudo, T., & Richardson, J. (2018). SentencePiece. EMNLP.

Related Models

• TinyTurk 20M (multi-domain): stunmuffin/tinyturk-v2.7d-mid-en
• TinyTurk English agent: stunmuffin/tinyturk-v2.7d-agent-en
• TinyTurk Turkish agent: stunmuffin/tinyturk-v2.7d-agent

Summary

TinyTurk v2.7d FineWeb 40M is a 42,004,992-parameter English causal language model trained from scratch on a 2.99-million-sample training split of FineWeb-Edu (approximately 765.44 million training tokens, seed 42, single run).

Under the EleutherAI Language Model Evaluation Harness (lm-eval 0.4.5, 0-shot, cache disabled), the model obtains:

TinyTurk 42M is consistently below GPT-2 Small across all six benchmarks under an identical pipeline, with an unweighted mean gap of -0.0406 accuracy points. The parameter-count difference (42M vs 124M) may contribute substantially to this gap, but the comparison does not isolate parameter count from other model differences.