NanoI1 is a 155M-parameter decoder-only language model trained from scratch by KiteFish AI on ~1.0T tokens. It is built for low-cost, on-premise deployment in Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction, classification and grounded Q&A, where data residency and inference cost matter more than open-ended general knowledge.
This repository holds the base (pretrained) model, best used as a starting point for fine-tuning. For chat and instruction following use KiteFishAI/NanoI1-155M-Instruct.
| Parameters | 155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection |
| Architecture | Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE |
| Training tokens | ~998B (pretraining 629B + extension 319B + anneal 50B) |
| Context length | 2,048 tokens (trained) |
| Tokenizer | Byte-level BPE, 50,304 tokens, trained on our own mixture |
| Languages | English; Hindi and 8 other Indian languages (limited) |
| Licence | Apache-2.0 |
The model uses a custom architecture, so loading needs the Python files in this repository (no transformers
class yet).
pip install torch safetensors tokenizers huggingface_hub
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("KiteFishAI/NanoI1-155M")
sys.path.insert(0, path)
import torch
from modeling_nanoi1 import from_pretrained
from chat import generate
model, tok = from_pretrained(path)
ids = [tok.token_to_id("<|endoftext|>")] + tok.encode("The Reserve Bank of India regulates", add_special_tokens=False).ids
out = generate(model, ids, max_new_tokens=40, temperature=0) # greedy continuation
print(tok.decode(out))
To fine-tune, load the weights the same way and train with your own loop, or use sft.py from the training
repository with the original checkpoint.
| Hyper-parameter | Value |
|---|---|
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (head dim 64) |
| KV latent dimension (MLA) | 256 |
| FFN | SwiGLU, 2,048 hidden |
| Normalisation | Pre-norm RMSNorm (ε = 1e-6), QK-norm |
| Position encoding | RoPE, θ = 5,000,000 |
| Embeddings | Untied input / output |
| Vocabulary | 50,304 (byte-level BPE, 32 special tokens) |
Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values (2 × 768), so the KV cache is 6× smaller than standard multi-head attention at the same width.
| Stage | Tokens | Data | LR schedule |
|---|---|---|---|
| 1. Pretraining | 629B (steps 0–600k) | Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia | linear warmup, then constant 6e-4 |
| 2. Extension | 319B (600k–904k) | Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B) | constant 6e-4 |
| 3. Anneal | 50B (904k–952k) | High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources | linear decay 6e-4 → 0 |
Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better).
| Benchmark | Shots | NanoI1-155M | Gemma-3-270M | Qwen3-0.6B-Base |
|---|---|---|---|---|
| WinoGrande | 5 | 52.6 | 54.0 | 59.9 |
| ARC-Easy | 25 | 58.9 | 59.0 | 74.9 |
| ARC-Challenge | 25 | 28.1 | 30.0 | 45.1 |
| OpenBookQA | 0 | 32.2 | 30.6 | 34.6 |
| RACE-High | 5 | 33.1 | 31.4 | 37.3 |
| TriviaQA | 5 | 10.2 | 14.3 | 20.1 |
| NaturalQuestions | 5 | 3.2 | 4.1 | 7.6 |
| MMLU-Redux | 5 | 27.8 | 28.2 | 32.1 |
| MMLU-Pro | 5 | 7.8 | 8.5 | 25.5 |
| GPQA | 5 | 26.6 | 25.9 | 29.9 |
| BBH | 3 | 18.4 | 22.7 | 39.4 |
| GSM8K | 8 | 4.0 | 1.8 | 48.9 |
| MATH | 4 | 2.2 | 2.4 | 29.9 |
| HumanEval | 0 | 9.1 | 2.4 | 29.3 |
| EvalPlus | 0 | 4.3 | 1.2 | 13.0 |
| INCLUDE (44 languages) | – | 26.0 | 25.5 | 36.3 |
| MMMLU | 5 | 23.6 | 25.6 | 38.7 |
| Pile test (BPB ↓) | – | 0.870 | 0.868 | 0.716 |
Reading the table:
All third-party datasets were used under their published terms:
See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated by other language models.
@techreport{kitefish2026nanoi1,
title = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries},
author = {Anuj and {KiteFish AI}},
institution = {KiteFish AI},
year = {2026}
}