The entire model is smaller than some README files.
The World's Tiniest Transformer
Big language models chase billions of parameters.
NanoZephyr went the opposite direction.
With only 372 parameters, this tiny GPT-2 style model explores just how absurdly small a language model can become while still technically generating text.
It's not smart.
It was never supposed to be.
Why?
Sometimes research starts with one simple question.
"How small can we make a Transformer before it becomes completely ridiculous?"
NanoZephyr is our answer.
- Built for fun.
- Built for curiosity.
- Built to make AI researchers laugh.
Specifications
| Property | Value |
|---|---|
| Parameters | 372 |
| Architecture | GPT-2 Style |
| Layers | 1 |
| Attention Heads | 1 |
| Embedding Size | 4 |
| Feed Forward | 8 |
| Vocabulary | 32 Tokens |
| Context Length | 16 |
| Model Size | ~15 KB |
| Training | CPU |
Example
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(
"WhirlwindAI/NanoZephyr"
)
model = AutoModelForCausalLM.from_pretrained(
"WhirlwindAI/NanoZephyr"
)
prompt = "The future of AI"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_length=16,
do_sample=True,
temperature=2.0
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Example output
The future of AI vxbzq rpfm lo...
Beautiful.
Live System Status
┌───────────────────────────────┐
│ NanoZephyr Boot Sequence │
├───────────────────────────────┤
│ Parameters : 372 │
│ GPU Usage : Basically none │
│ Intelligence : █░░░░░░░░ 3% │
│ Confidence : ██████████ 100%│
│ Randomness : ██████████ MAX │
│ Status : ONLINE │
└───────────────────────────────┘
Performance
| Benchmark | Result |
|---|---|
| Common Sense | 0.01 |
| Mathematics | 0.00 |
| Philosophy | ??? |
| Gibberish | 100.00 |
| Comedy | ∞ |
Sample Outputs
Input:
hello
Output:
helloclvtdzng o
Input:
ROMEO:
Output:
ufbgdyo zia
Input:
Once upon a time...
Output:
qxwwbbvh zjv
Every generation is a surprise.
Sometimes even to the model.
Intended Uses
✅ Learning how Transformers work
✅ Educational demos
✅ Parameter-count experiments
✅ AI memes
✅ Making your 70B model feel better
Not Intended For
❌ Homework
❌ Medical advice
❌ Programming
❌ Legal documents
❌ Anything requiring intelligence
Awards
🥇 Smallest Transformer (probably)
🥇 Highest Gibberish Density
🥇 Lowest Storage Requirement
🥇 Fastest CPU Training
🏆 Most Honest AI Model
License
MIT
Feel free to use it, modify it, laugh at it, or make it even smaller.