Nova-MoE (122M Total / 34.5M Active)
Nova-MoE is a sparse Mixture-of-Experts (MoE) causal language model optimized for parameter-efficient compute scaling and real-time narrative generation on the TinyStories benchmark.
- GitHub Repository & Full Source Code: https://github.com/sarimahsan/nova-moe
- Hugging Face Model Hub: https://huggingface.co/sarimahsan101/nova-mooe
It combines state-of-the-art architectural advancements---RMSNorm, Rotary Position Embeddings (RoPE), Grouped-Query Attention (GQA), and SwiGLU Feed-Forward Networks---with Expert-Choice Routing (Zhou et al., 2022).
🚀 How to Run Inference
To run story generation with this model, clone the repository and run generate.py:
# 1. Clone the repository
git clone https://github.com/sarimahsan/nova-moe.git
cd nova-moe
# 2. Install dependencies
pip install -r requirements.txt
# 3. Generate story using model weights from Hugging Face
python generate.py --hf_repo sarimahsan101/nova-mooe --prompt "Once upon a time, there was a little girl named Lily"
For full architecture details, model benchmarks, and training scripts, visit the GitHub Repository.
📊 Model Highlights
- Total Parameters (Disk): 122,610,176 (~122.6 Million)
- Active Parameters (Per Token): 34,529,792 (~34.5 Million)
- Active Parameter Ratio: 28.2% (71.8% per-token FLOP reduction)
- Inference Throughput: 58.8 tokens/sec on NVIDIA Tesla T4 GPU
- Token Latency: 17.02 ms/token
- VRAM Footprint: 4.2 GB FP16 memory
🏗️ Architecture Specifications
| Attribute | Specification |
|---|---|
| Source Code | github.com/sarimahsan/nova-moe |
| Hidden Dimension ($d_{\text{model}}$) | 512 |
| Total Layers ($L$) | 8 Transformer Blocks |
| Query Attention Heads ($H_Q$) | 8 |
| Key-Value Attention Heads ($H_{KV}$) | 2 (Grouped-Query Attention, 4x compression) |
| Head Dimension ($d_{\text{head}}$) | 64 |
| Positional Embedding | Rotary Position Embeddings (RoPE) |
| MoE Layers | Layers 1, 3, 5, 7 (4 MoE Layers) |
| Experts per Layer ($E$) | 8 Experts per MoE Layer |
| Expert Routing Algorithm | Expert-Choice Routing (Column Softmax over Tokens) |
| Expert Capacity Factor ($\alpha$) | 1.25 |
| Dense FFN / Expert Dim ($d_{\text{ffn}}$) | 2,048 (SwiGLU) |
| Vocabulary Size ($V$) | 8,000 (Byte-Level BPE Tokenizer) |
📈 Empirical Benchmarks (NVIDIA Tesla T4 GPU)
| Metric | Measured Result |
|---|---|
| Inference Precision | PyTorch AMP FP16 |
| Generation Throughput | 58.8 tokens / sec |
| Per-Token Latency | 17.02 ms / token |
| Model Weight VRAM | 4,228.8 MB |
| Peak Generation VRAM | 4,239.3 MB |
📝 Sample Generated Story
Prompt: "Once upon a time, there was a little girl named Lily"
Output: "Once upon a time, there was a little girl named Lily. She was a good girl and always listened to her mom. One day, Lily's mom asked her to clean up her toys. Lily didn't want to because she was having too much fun. But her mom said, 'If you don't clean up, you won't be able to play with your toys again.'
Lily didn't want to clean up her toys because she was having too much fun playing with them. But then she remembered that she loved playing with her toys. She decided to clean up her toys and then went to play with her mom.
After she finished cleaning, Lily felt happy that she was able to play with her toys without her mom making a mess. She learned that it's important to keep clean."
📜 Citation
If you use Nova-MoE in your research or project, please cite:
@article{ahsan2026novamoe,
title={Nova-MoE: An Efficient Sparse Mixture-of-Experts Language Model with Expert-Choice Routing},
author={Ahsan, Sarim},
url={https://github.com/sarimahsan/nova-moe},
year={2026},
publisher={Hugging Face},
journal={Hugging Face Model Hub}
}