Nano.Deep.Reasoner.11m

A small decoder-only causal language model trained from scratch.

Architecture

• Parameters: 11,229,120
• Context length: 1096
• Hidden size: 360
• Layers: 6
• Attention heads: 8
• Intermediate size: 1440
• Vocabulary size: 4,096

Reasoning tokens

• <|input|>
• <|think|>
• <|thought|>
• <|reasoning|>
• <|answer|>

Objective

True causal next-token prediction.

For an input sequence:

x[0], x[1], x[2], ...

the model learns:

x[0] -> x[1]

x[1] -> x[2]

x[2] -> x[3]

and so on.

Context

Every individual training example is strictly limited to:

1095 content tokens + EOS

and padded to exactly 1096 positions.

No oversized example is intentionally split across separate training examples.

Example selection

Starting from Session 3, new examples are selected via a deterministic shuffled scan of the dataset (seeded, reproducible across runs) rather than raw sequential order, to avoid category/source concentration within a session. Sessions 1-2 (40,000 examples) were selected sequentially before this correction and remain part of the trained corpus.

Dataset

Plans11/Organized_PreTrain_1k_Context

Categories exposed by the dataset include:

• Think
• Thought
• Reasoning
• Instruct
• Chat
• Tool_Calling
• Code_Instruct

Resumability

Training state is persisted to Hugging Face, including:

• model.safetensors
• training_state.pt
• tokenizer files
• config.json
• progress.json
• seen_examples.jsonl
• training_metadata.json

Examples are tracked using SHA-256 content hashes.

The tokenizer becomes immutable after its initial creation.

Current progress

• Unique examples trained: 60,000
• Global optimizer steps: 1,441
• Last session: 20,000

This is an experimental small language model and is not guaranteed to produce factually or logically correct outputs.