The Idea
Bigger isn't always better.TinyZephyr explores the opposite extreme. Instead of billions of parameters, this model asks one very important question: How ridiculously small can a transformer become before it completely loses its mind? The answer is... surprisingly entertaining. |
Why?
Nobody needed this.
Nobody requested it.
Nobody funded it.
Yet somehow...
TinyZephyr exists.
It was built purely as an experiment to explore the lower limits of transformer architectures while proving that even microscopic language models deserve beautiful documentation.
Specifications
| Property | Value |
|---|---|
| Parameters | 1,272 |
| Architecture | GPT-2 |
| Layers | 1 |
| Attention Heads | 1 |
| Embedding Size | 8 |
| Context Length | 32 |
| Vocabulary | 50 Tokens |
| Model Size | ~25 KB |
| Training Time | ~4 Minutes (CPU) |
Benchmark
| Task | Result |
|---|---|
| Write Python | ❌ |
| Solve Math | ❌ |
| Explain Physics | ❌ |
| Generate Gibberish | ✅ |
| Exist | ✅ |
Quick Start
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("WhirlwindAI/TinyZephyr")
model = AutoModelForCausalLM.from_pretrained("WhirlwindAI/TinyZephyr")
prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
do_sample=True,
temperature=1.6,
max_length=32
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Possible output:
The meaning of life is...
xqw fjczqnv lpoqa yv
Beautiful.
Example Conversation
You
Write a poem about space.
TinyZephyr
moon potato quantum fish sandwich
Mission accomplished.
System Requirements
CPU
Yes.
GPU
Optional.
RAM
If your browser opens, you're probably fine.
Storage
Less than most PNG files.
Frequently Asked Questions
Is this useful?
Not particularly.
Is this serious research?
Surprisingly... yes.
Can it replace ChatGPT?
Only if your expectations are extremely flexible.
Why did you build this?
Curiosity.
And because somebody had to.
Awards
🏆 Fastest Model To Finish Inference
🥇 Most Parameters Removed Without Deleting Everything
🥈 Best Random Sentence Generator
🥉 Self-Proclaimed Champion of Tiny AI
Limitations
TinyZephyr was never trained to be helpful.
It doesn't know facts.
It doesn't reason.
It doesn't write code.
It mostly produces beautifully random nonsense.
And that's exactly what it was designed to do.
License
MIT
Use it.
Benchmark it.
Laugh at it.
Make it even smaller.
Built by WhirlwindAI
Sometimes the best experiments begin with terrible ideas.