TinyStories-Kilat-35M ⚡

A high-efficiency 35M-parameter sub-quadratic causal language model powered by the Kilat Macro-Synergy Hybrid Dense architecture.

TinyStories-Kilat-35M is a next-generation lightweight language model trained from scratch on the TinyStories dataset. It explores sub-quadratic sequence modeling at a compact scale, combining Macro-Synergy Hybrid Topology ($O(1)$ constant-memory Global Exponential Decay on even layers + $93.75%$ compressed Decoupled RoPE Latent MLA on odd layers) with a Dense SwiGLU feed-forward network.

The model is developed and certified using the Kilat AI Framework — an advanced PyTorch & Hugging Face ecosystem toolkit engineered for sub-quadratic pre-training, mathematical pre-flight assurance, and high-throughput inference.

Project Links & Resources

• Core Engine: Airukua/Kilat (GitHub)
• Model Creator: AiRukua (Hugging Face)
• Training Corpus: roneneldan/TinyStories

---

⚡ Key Specifications

---

🔬 Architectural Topology: Macro-Synergy Dense Hybrid

Standard Transformer models scale quadratically $O(N^2)$ with sequence length, causing severe KV-cache memory bloat during inference. TinyStories-Kilat-35M eliminates this bottleneck by deploying an interleaved dual-pathway architecture with dense SwiGLU projections:

Input Tokens
     │
     ▼
┌────────────────────────────────────────────────────────┐
│  Layer 0 (Even) : Global Exponential Decay (O(1) State)│
│  FFN Block       : Dense SwiGLU (Gate / Up / Down)     │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│  Layer 1 (Odd)  : Decoupled RoPE Latent MLA (KV 96-dim)│
│  FFN Block       : Dense SwiGLU (Gate / Up / Down)     │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
               [Layers 2–7 Interleaved]
                           │
                           ▼
                   Output Logits

1. Global Exponential Decay (Even Layers: 0, 2, 4, 6)

Operates as a linear-time recurrent memory mechanism with an $O(1)$ constant-size hidden state: $$s_t = \text{diag}(\lambda) s_{t-1} + v_t$$ Even layers require zero KV-cache, providing instantaneous prompt prefill and constant memory consumption regardless of document length.

2. Multi-Head Latent Attention (MLA) with Decoupled RoPE (Odd Layers: 1, 3, 5, 7)

Compresses key and value projections into a compact low-rank latent vector: $$c_t^{KV} = W^{DKV} x_t, \quad d_c = 96$$ Instead of caching individual key and value vectors across all heads, only the 96-dimensional latent vector $c_t^{KV}$ is retained in memory ($93.75%$ cache compression). Precise associative retrieval and relative token positions are maintained via Decoupled Rotary Position Embedding (RoPE).

3. Dense SwiGLU Feed-Forward Network

Each block deploys a dense Gated Linear Unit with SiLU (Swish) activation: $$\text{FFN}(x) = \left(\text{swish}(x W_{\text{gate}}) \odot x W_{\text{up}}\right) W_{\text{down}}$$ With an intermediate expansion factor of $2.67\times$ ($d_{\text{ffn}} = 1,024$), this provides dense expressive capacity without routing overhead.

---

🚀 Quickstart & Inference

Option 1: Standard Hugging Face transformers (Zero Kilat Installation)

Thanks to standalone remote code mapping (auto_map), you can load and run TinyStories-Kilat-35M directly using standard Hugging Face pipelines without installing Kilat:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_id = "AiRukua/tiny-stories-kilat-35M"

# 1. Load tokenizer and model directly from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
)

# 2. Real-time token-by-token live streaming generation
prompt = "Once upon a time, Lily found a tiny kitten"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
streamer = TextStreamer(tokenizer, skip_prompt=True)

print(f"Prompt: {prompt}\n\nGenerated: ", end="", flush=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    streamer=streamer,
)

---

Option 2: Native Kilat Engine (Optimized Low-Latency Streaming)

Install the Kilat framework:

pip install git+https://github.com/Airukua/Kilat.git safetensors transformers

Run inference with native low-latency generator streaming:

import torch
from transformers import AutoTokenizer
from kilat.models import KilatTransformer
from kilat.inference import TextGenerator

model_id = "AiRukua/tiny-stories-kilat-35M"

device = "cuda" if torch.cuda.is_available() else "cpu"
model = KilatTransformer.from_pretrained(model_id).to(device)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model.eval()

generator = TextGenerator(model, tokenizer)
prompt = "Once upon a time, Lily found a tiny kitten"

print(f"Prompt: {prompt}\n\nGenerated: ", end="", flush=True)
for chunk in generator.stream(prompt, max_new_tokens=100, temperature=0.7, top_k=40):
    print(chunk, end="", flush=True)
print("\n")

---

Option 3: Terminal CLI Text Generation

kilat generate \
  --checkpoint "AiRukua/tiny-stories-kilat-35M" \
  --prompt "Once upon a time, Lily found a tiny kitten" \
  --max-new-tokens 128 \
  --temperature 0.7 \
  --top-p 0.9

---

Option 4: Deploy OpenAI-Compatible REST API Server

Launch a high-throughput OpenAI-compatible API endpoint supporting Server-Sent Events (SSE) streaming:

kilat serve --checkpoint "AiRukua/tiny-stories-kilat-35M" --port 8000

Connect using standard OpenAI Python SDK or ChatGPT frontends (LibreChat, Open WebUI, Cursor):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="kilat")

response = client.chat.completions.create(
    model="AiRukua/tiny-stories-kilat-35M",
    messages=[{"role": "user", "content": "Once upon a time, Lily found a tiny kitten"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

---

📏 Context Window Extension via YaRN

TinyStories-Kilat-35M natively supports YaRN (Yet another RoPE extensioN) on its Latent MLA attention layers. You can dynamically scale the context window at inference time without fine-tuning:

from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("AiRukua/tiny-stories-kilat-35M", trust_remote_code=True)
# Extend context 4x (from 512 tokens to 2,048 tokens)
config.yarn_scale = 4.0
config.yarn_base_seq_len = 512

model = AutoModelForCausalLM.from_pretrained(
    "AiRukua/tiny-stories-kilat-35M",
    config=config,
    trust_remote_code=True,
)

---

🛠️ Post-Training & Ecosystem Bridges

TinyStories-Kilat-35M is natively compatible with standard post-training libraries:

• PEFT / LoRA: Wrap and fine-tune dense linear layers or MLA projections using get_peft_model(model, LoraConfig(...)).
• TRL (SFT & DPO): Fine-tune instruction models using SFTTrainer with KilatCompletionOnlyCollator, or align preferences via DPOTrainer.
• Assurance Audit: Run 7-gate static and dynamic architecture audits anytime using kilat verify --model AiRukua/tiny-stories-kilat-35M.

---

📊 Example Generations

Prompt 1

Once upon a time, Lily found a tiny kitten.

Model Output

The kitten was very small and cute. Lily wanted to keep the kitten, but she knew it was not a good toy. So, she decided to take the kitten home with her. She gave it some milk and made a soft bed in a box. Soon, the little kitten began to purr happily, and Lily smiled with joy.

Prompt 2

One sunny morning, Tim and his puppy

Model Output

went to the park. They saw a big tree with a swing. Tim wanted to swing on the swing, but he was too small. His mom said, "Let's try to balance on the swing." Tim tried to balance on the swing, and soon he was laughing and having fun with his puppy.

---

🛡️ Pre-Flight Assurance & Quality Certification

This model was trained and certified under the AGENTS.md Kilat Assurance Standard:

• Gate 0 (Security & Deserialization): Clean SafeTensors weights (zero arbitrary code execution / pickle vulnerability).
• Gate 1 (Theoretical Invariants): Residual stream variance strictly bounded at $O(\sqrt{L})$ with zero NaN/Inf gradients.
• Gate 2 (Numerical Stability): Output logits verified bounded under mixed-precision FP16/BF16 arithmetic.
• Gate 3 (State Cache Equivalence): Bitwise numerical equivalence between full sequence prefill and single-token recurrent decoding.
• Gate 4 (Causal Integrity): Autoregressive mask strictly enforced (zero future-token attention leakage).
• Gate 5 (Optimization Sanity): Decoupled weight decay groups isolating 1D normalization and recurrent decay parameters.

---

📜 Citation & License

The model weights and code are released under the MIT License.

If you use TinyStories-Kilat-35M or the Kilat architecture in your research or applications, please cite:

@software{kilat2026,
  author = {AiRukua},
  title = {Kilat: High-Efficiency Sub-Quadratic Hybrid Language Model Framework},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Airukua/Kilat}}
}

Created with ⚡ by AiRukua
Accelerating open-source AI with sub-quadratic architectures.