🍡 Mochi-Novel-300M: Dual-Engine Storytelling LLM Built 100% From Scratch in PyTorch
Mochi-Novel-300M is a 295.1 million parameter modern Large Language Model designed, implemented, pre-trained, and fine-tuned 100% from scratch in pure PyTorch—with zero dependencies on Hugging Face transformers modeling code.
It incorporates Rotary Position Embeddings (RoPE), RMSNorm, SwiGLU feed-forward networks, Key-Value (KV) Caching, and a specialized Dual-Engine Architecture for long-form, multi-scene novel generation.
🏛️ Model Specifications
| Hyperparameter | Value |
|---|---|
| Total Parameters | 295,084,032 (295.1M) |
| Transformer Blocks | 24 Layers |
| Attention Heads | 16 Heads (Head Dim: 64) |
| Embedding Dimension ($d_{model}$) | 1,024 |
| SwiGLU Hidden Dimension | 2,730 |
| Vocabulary Size | 16,384 (Custom SentencePiece BPE) |
| Context Window ($T$) | 1,536 Tokens |
| Positional Encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm (Pre-LayerNorm) |
| Activation | SwiGLU Gated Linear Unit |
| Inference Generation Speed | ~44.5 tokens/sec (NVIDIA RTX 3060) |
| Decoding Protection | Strict 4-Gram Blocking + Repetition Penalty (1.15) |
🎭 The Dual-Engine System
mochi_novel_300m_golden_rajput.pt(Golden Novel Engine):- Fine-tuned for published-grade dramatic literature.
- Captures character voice, dialogue tension, and multi-scene continuity.
mochi_novel_300m.pt(Universal Multi-Genre Engine):- Aligned across 20+ diverse literary genres (Sci-Fi, Cyberpunk, Gothic, Noir, Fantasy, Horror, Western).
- 90% concept leak-free rate across unrelated universes.
🚀 Quickstart & Inference
git clone https://github.com/shashankswaroop789-eng/mochi-88m.git
cd mochi-88m
python download_weights.py
python story_studio_gui.py
```---
language:
- en
license: mit
tags:
- pytorch
- text-generation
- storytelling
- causal-lm
- creative-writing
- dual-engine
- scratch-built
pipeline_tag: text-generation
---
# 🍡 Mochi-Novel-300M: Dual-Engine Storytelling LLM Built 100% From Scratch in PyTorch
**Mochi-Novel-300M** is a 295.1 million parameter modern Large Language Model designed, implemented, pre-trained, and fine-tuned **100% from scratch in pure PyTorch**—with zero dependencies on Hugging Face `transformers` modeling code.
It incorporates **Rotary Position Embeddings (RoPE)**, **RMSNorm**, **SwiGLU feed-forward networks**, **Key-Value (KV) Caching**, and a specialized **Dual-Engine Architecture** for long-form, multi-scene novel generation.
---
## 🏛️ Model Specifications
| Hyperparameter | Value |
| :--- | :--- |
| **Total Parameters** | **295,084,032 (295.1M)** |
| **Transformer Blocks** | **24 Layers** |
| **Attention Heads** | **16 Heads** (Head Dim: 64) |
| **Embedding Dimension ($d_{model}$)** | **1,024** |
| **SwiGLU Hidden Dimension** | **2,730** |
| **Vocabulary Size** | **16,384** (Custom SentencePiece BPE) |
| **Context Window ($T$)** | **1,536 Tokens** |
| **Positional Encoding** | Rotary Position Embeddings (RoPE) |
| **Normalization** | RMSNorm (Pre-LayerNorm) |
| **Activation** | SwiGLU Gated Linear Unit |
| **Inference Generation Speed** | **~44.5 tokens/sec** (NVIDIA RTX 3060) |
| **Decoding Protection** | Strict 4-Gram Blocking + Repetition Penalty (1.15) |
---
## 🎭 The Dual-Engine System
1. **`mochi_novel_300m_golden_rajput.pt` (Golden Novel Engine):**
- Fine-tuned for published-grade dramatic literature.
- Captures character voice, dialogue tension, and multi-scene continuity.
2. **`mochi_novel_300m.pt` (Universal Multi-Genre Engine):**
- Aligned across 20+ diverse literary genres (Sci-Fi, Cyberpunk, Gothic, Noir, Fantasy, Horror, Western).
- 90% concept leak-free rate across unrelated universes.
---
## 🚀 Quickstart & Inference
```bash
git clone https://github.com/shashankswaroop789-eng/mochi-88m.git
cd mochi-88m
python download_weights.py
python story_studio_gui.py