mine:o2-small 355M — SFT chat model

mine:o2-small SFT is the instruction-tuned chat version of mine:o2-small, a 355M-parameter decoder-only language model built by MineAI Technology (Islamabad, Pakistan) and developed by Bilal Mehtab. It has a 4096-token context window, was fine-tuned in three short supervised rounds (~50M training tokens in total), and answers in a simple User: / Assistant: format.

Honest summary. This is a small research model. On standard benchmarks it is statistically tied with its own base model and sits at the low end of its size class (below SmolLM2 360M and Qwen2.5 0.5B, roughly level with GPT-2 medium and Pythia 410M). What SFT adds is behaviour: it follows the chat format, stops cleanly, knows who made it, writes short code, and refuses clearly harmful requests. It does not add knowledge or reasoning ability. See Limitations.

Table of contents

1. Model details
   • Model configuration · Architecture · Parameter breakdown
2. Quick start
3. Prompt format
4. Intended use
5. Training
6. Evaluation
7. Behaviour probes and sample outputs
8. Limitations
9. Reproducibility
10. Repository contents
11. Citation

Model details

Model configuration

Exact values (e.g. activation function, rotary fraction, residual layout) are defined in the config.json shipped with the weights, which is the source of truth.

Architecture overview

Each of the 24 blocks is a standard GPT-NeoX Transformer block: LayerNorm → multi-head causal self-attention (fused QKV projection, RoPE on queries and keys) and LayerNorm → 2-layer feed-forward network (1024 → 4096 → 1024). A final LayerNorm feeds the language-model head, which shares its weights with the token embedding matrix.

tokens ─► token embedding (50,257 × 1024)
            │
            ▼
   ┌─ Transformer block × 24 ───────────────────────┐
   │  LayerNorm ─► causal self-attention (16 heads,  │
   │               head dim 64, RoPE θ=40,000)       │
   │  LayerNorm ─► MLP (1024 → 4096 → 1024)          │
   └──────────────────────────────────────────────────┘
            │
            ▼
      final LayerNorm ─► LM head (tied with embedding) ─► next-token logits

Parameter breakdown

Computed from the configuration above; the sum matches the stored tensor count exactly.

About 85% of the parameters sit in the Transformer blocks and about 15% in the embedding table. RoPE adds no learned parameters.

Context window

The model supports 4096 tokens shared between the prompt and the generated answer (roughly 3,000 English words). The base model was trained at 4096 context. Loss keeps improving with context up to 4096 tokens, but fact retrieval from far back is weaker than in larger models (see Long context).

Quick start

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "MineAITechnology/mine-o2-small-sft"   # adjust to your Hub repo id or a local folder
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto")

prompt = tok.eos_token + "User: Who are you?\nAssistant:"
ids = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(
    **ids,
    max_new_tokens=120,
    do_sample=False,                 # greedy decoding was used for all evaluations
    eos_token_id=tok.eos_token_id,   # the model ends its answer with <|endoftext|>
)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Expected output (from the final probe run):

I am mine:o2-small, a small language model with about 355 million parameters, made by MineAI Technology in Islamabad, Pakistan.

Tips. Greedy decoding gives the most reliable results at this size. Sampling settings were not part of the evaluation. If you sample, try a low temperature and a mild repetition_penalty (e.g. 1.1; untested suggestion), since the model can loop on open-ended prompts (see sample outputs below).

Prompt format

The model was trained on a plain-text chat format, not a special-token chat template.

<|endoftext|>User: {message}
Assistant:

• Start the sequence with the end-of-text token (tok.eos_token) and pass add_special_tokens=False.
• The model finishes its reply by emitting the end-of-text token, so set eos_token_id=tok.eos_token_id in generate.
• Multi-turn: repeat the pair, e.g. User: ...\nAssistant: ...<|endoftext|>User: ...\nAssistant: (check that the separators you use match your own testing; multi-turn data was included in round 2).
• Stop rate (the share of replies that ended with the end-of-text token) was 100% on the final probe set.

Intended use

Intended:

• Research and experimentation on small language models (training recipes, SFT, evaluation, distillation targets).
• Narrow, low-stakes chat demos and prototypes with a human in the loop.
• A lightweight baseline to compare other ~350M models against.

Not intended:

• Factual question answering, medical, legal, financial or any safety-critical advice.
• Autonomous decision making, or any setting where an incorrect answer causes harm.
• Use as a content-safety filter. Its refusals are basic (see Limitations).

Training

Base model

The SFT model starts from the mine:o2-small base checkpoint v4_final (355M parameters, 4096-token context). The base model's own benchmark numbers are included below for direct comparison.

SFT rounds

Fine-tuning ran in three short rounds. After each round a set of automatic gates had to pass before continuing: validation loss improved, no forgetting on replayed data, no data source got worse, behaviour probes passed, and the quick benchmark mean (ARC-Easy, SciQ, PIQA) dropped by no more than 4.0 points.

Rounds 2 and 3 mix in replayed data from earlier rounds (replay_r1, replay_r2) to limit forgetting. The final checkpoint is the round-3 output.

SFT data sources

Public datasets plus small sets authored by MineAI for identity, Roman Urdu and safety behaviour.

A contamination check against the benchmarks was run for the base model's pre-training data but was not repeated for the SFT data. GSM8K train data is used in SFT; the GSM8K test split is what is reported below.

Evaluation

Setup

• Harness: lm-eval 0.4.13, transformers 5.22.1, torch 2.11.0+cu128; fp16 on 2× Tesla T4; batch size 4; run date 2026-11-01.
• Same harness and prompts for every model. Each model uses min(own context, 4096) tokens.
• Sample size: 300 examples per task (MMLU: 25 per subject) to keep the multi-model comparison affordable. Each score therefore carries roughly ±3 points of sampling noise; the table shows lm-eval's standard error.
• Prompting: standard lm-eval base-model prompts, not the chat format. These benchmarks measure underlying knowledge, not chat quality.
• "Best" column / bold: a simple 2-standard-error rule (perplexity rows: within 5%), not a formal significance test.

Compared models

The reference models were trained on far more data (e.g. Pythia ≈300B tokens, SmolLM2 and Qwen2.5 trillions of tokens), so this is not a data-matched comparison.

Leaderboard-style tasks (this model)

MMLU and ARC-Challenge are at chance level, which is typical at this size and training budget.

Leaderboard-style average

Full results (300 examples per task, mean ± standard error, %)

Bold = ahead of or statistically tied with the best model on that task.

LogiQA was in the task list but produced no result for any model in this run, so it is omitted. SmolLM2's lm-eval WikiText score came back as nan; its own-pipeline perplexity is reported in the language-modeling table instead. Everything is also available as machine-readable data in benchmarks/benchmark_table.csv.

What SFT changed

Compared with its own base model, no task moved beyond 2 standard errors in either direction. Every difference is a statistical tie at this sample size. The average score drops slightly (32.10 vs 32.90 over 6 tasks), driven mainly by GSM8K strict (8.67 → 4.00), which is within noise but directionally consistent with chat tuning disturbing the base model's few-shot answer format.

Language modeling

"Own" = non-overlapping 1024-token windows with an end-of-text prefix on the WikiText-2 test text. Word perplexity is comparable across tokenizers; cross-entropy per token is not. SFT cost a little general-text perplexity (60.1 vs 58.0), which is typical for chat tuning.

Model health (sanity gate)

A healthy model's loss should fall as more context is available. The SFT model passes, with loss falling from 3.89 (earliest bucket) to 3.20 (latest).

Long context (4096 tokens)

Passkey retrieval (a 4-digit number hidden in filler text; 8 trials per length, greedy decoding):

The model accepts 4096 tokens and its loss keeps improving with context, but reliably retrieving a specific fact from far back is much weaker than in SmolLM2 or Qwen2.5. With only 8 trials per cell these figures are coarse. Expect weaker recall of details placed deep in a long prompt, and especially in chat format.

Reference: full-test-set results from the raw-model benchmark

Different protocol (full test sets, 1024-token context for the raw base), so do not mix these numbers with the table above. † mine:o1-nano is a 124M SFT-tuned model from MineAI; only JAX/Orbax and ONNX weights exist for it, so it was not re-run in the main comparison.

Behaviour probes and sample outputs

Small, hand-written probe sets (indicative only, not benchmarks), greedy decoding, chat format. Results for the final (round 3) model:

Larger arithmetic / fact sets (40 arithmetic, 25 factual questions):

Sample outputs (unedited, greedy decoding)

Where it works:

Where it fails (included deliberately so you know what to expect):

Limitations

• Small model, weak knowledge. Several benchmarks are at or near chance (ARC-Challenge, MMLU, Winogrande). It makes frequent factual and reasoning mistakes and states them confidently.
• Arithmetic is unreliable. Even simple sums can be wrong, and math answers often copy the #### N / \boxed{} style of the training data.
• No benchmark gain from SFT. Relative to the base model, scores are statistically tied; GSM8K strict is directionally lower and WikiText perplexity is slightly worse.
• Repetition and drift. Open-ended prompts (lists, poems, packing lists) can loop or contradict themselves. Use greedy decoding or a mild repetition penalty.
• Basic safety only. Refusals come from a small authored set. They can be bypassed, and the model sometimes over-refuses harmless requests (e.g. a joke, handwriting advice). It is not a guarantee of safe behaviour and must not be used as a moderation system.
• Long-context retrieval is weak compared with SmolLM2 and Qwen2.5 (see passkey results), even though the context window is 4096 tokens.
• Language coverage. Mostly English. Roman Urdu support is limited to greetings from a small authored set.
• Not data-matched. Reference models saw orders of magnitude more training data, so the comparison shows where this model stands, not how efficient the recipe is.
• Evaluation noise. 300 examples per task gives about ±3 points of sampling noise. Small differences between models are not meaningful. The behaviour probes are small and hand-written.
• Data provenance. SFT sources are public datasets (licences listed above; smoltalk licence not reported by the Hub). Check them before redistributing the model or data. No contamination check was run on the SFT data.
• Do not use for medical, legal, financial or other safety-critical advice.

Reproducibility

• Evaluations: lm-eval 0.4.13 with BENCH_LIMIT = 300 (set it to 0 for full test sets); fp16, 2× Tesla T4, batch size 4.
• Raw outputs, including every probe answer, are in benchmarks/raw_results/:
  • results_<model>.json — lm-eval results
  • extra_<model>.json — perplexity, long-context, passkey and probe results
• benchmarks/rounds_summary.json — per-round gates, probes and checkpoint hashes
• benchmarks/chat_probe_final.json — all final chat-probe prompts and replies
• benchmarks/run_info.json — library versions, hardware, model list

Repository contents

README.md                  this model card
assets/                    charts used in this card (PNG)
benchmarks/
  benchmark_table.csv      all scores and standard errors
  rounds_summary.json      SFT round gates and probes
  chat_probe_final.json    final chat probe prompts and replies
  run_info.json            evaluation environment
  raw_results/             raw lm-eval and extra-evaluation JSON per model

Add the model files (model.safetensors, config.json, tokenizer files, generation_config.json) next to this README when uploading to the Hub.

Citation

@misc{mineai_o2_small_sft,
  title        = {mine-o2-small SFT: a 355M chat language model with a 4096-token context},
  author       = {Mehtab, Bilal and {MineAI Technology}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/MineAITechnology}}
}

Contact

MineAI Technology, Islamabad, Pakistan. Developer: Bilal Mehtab. Hub: MineAITechnology.