NanoI1 is a 155M-parameter decoder-only language model trained from scratch by KiteFish AI on ~1.0T tokens. It is built for low-cost, on-premise deployment in Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction, classification and grounded Q&A, where data residency and inference cost matter more than open-ended general knowledge.
This repository holds the instruction-tuned model. The base model is KiteFishAI/NanoI1-155M.
| Parameters | 155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection |
| Architecture | Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE |
| Training tokens | ~998B (pretraining 629B + extension 319B + anneal 50B), then SFT |
| Context length | 2,048 tokens (trained) |
| Tokenizer | Byte-level BPE, 50,304 tokens, trained on our own mixture |
| Languages | English; Hindi and 8 other Indian languages (limited) |
| Chat format | ChatML (<|im_start|>role ... <|im_end|>) |
| Licence | Apache-2.0 |
The model uses a custom architecture, so loading needs the Python files in this repository (no transformers
class yet).
pip install torch safetensors tokenizers huggingface_hub
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("KiteFishAI/NanoI1-155M-Instruct")
sys.path.insert(0, path)
from modeling_nanoi1 import from_pretrained, chat
model, tok = from_pretrained(path) # GPU if available, bf16 on GPU
print(chat(model, tok, [
{"role": "system", "content": "You are a concise assistant for a bank's operations team."},
{"role": "user", "content": "Explain what a bank's KYC process is in two sentences."},
], temperature=0))
| Hyper-parameter | Value |
|---|---|
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (head dim 64) |
| KV latent dimension (MLA) | 256 |
| FFN | SwiGLU, 2,048 hidden |
| Normalisation | Pre-norm RMSNorm (ε = 1e-6), QK-norm |
| Position encoding | RoPE, θ = 5,000,000 |
| Embeddings | Untied input / output |
| Vocabulary | 50,304 (byte-level BPE, 32 special tokens) |
Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values (2 × 768), so the KV cache is 6× smaller than standard multi-head attention at the same width.
| Stage | Tokens | Data | LR schedule |
|---|---|---|---|
| 1. Pretraining | 629B (steps 0–600k) | Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia | linear warmup, then constant 6e-4 |
| 2. Extension | 319B (600k–904k) | Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B) | constant 6e-4 |
| 3. Anneal | 50B (904k–952k) | High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources | linear decay 6e-4 → 0 |
| 4. SFT | 2 epochs | smol-smoltalk + Aya (English and 9 Indian languages) | 1e-4, cosine to 1e-5 |
Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better).
| Benchmark | Shots | NanoI1-155M | Gemma-3-270M | Qwen3-0.6B-Base |
|---|---|---|---|---|
| WinoGrande | 5 | 52.6 | 54.0 | 59.9 |
| ARC-Easy | 25 | 58.9 | 59.0 | 74.9 |
| ARC-Challenge | 25 | 28.1 | 30.0 | 45.1 |
| OpenBookQA | 0 | 32.2 | 30.6 | 34.6 |
| RACE-High | 5 | 33.1 | 31.4 | 37.3 |
| NaturalQuestions | 5 | 3.2 | 4.1 | 7.6 |
| MMLU-Redux | 5 | 27.8 | 28.2 | 32.1 |
| MMLU-Pro | 5 | 7.8 | 8.5 | 25.5 |
| GPQA | 5 | 26.6 | 25.9 | 29.9 |
| BBH | 3 | 18.4 | 22.7 | 39.4 |
| GSM8K | 8 | 4.0 | 1.8 | 48.9 |
| MATH | 4 | 2.2 | 2.4 | 29.9 |
| HumanEval | 0 | 9.1 | 2.4 | 29.3 |
| EvalPlus | 0 | 4.3 | 1.2 | 13.0 |
| INCLUDE (44 languages) | – | 26.0 | 25.5 | 36.3 |
| MMMLU | 5 | 23.6 | 25.6 | 38.7 |
| Pile test (BPB ↓) | – | 0.870 | 0.868 | 0.716 |
Reading the table:
| Benchmark | NanoI1-155M-Instruct | SmolLM2-135M-Instruct | SmolLM2-360M-Instruct | Qwen2.5-0.5B-Instruct |
|---|---|---|---|---|
| IFEval (prompt-level loose) | [fill after run_sft.sh eval] |
All third-party datasets were used under their published terms:
See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated by other language models.
@techreport{kitefish2026nanoi1,
title = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries},
author = {Anuj and {KiteFish AI}},
institution = {KiteFish AI},
year = {2026}
}