🎼 ORCH Next.js 3B

A 3 billion parameter decoder-only transformer trained from scratch for generating complete Next.js applications — pages, API routes, Prisma schemas, Tailwind components, configs.

---

TL;DR

What this is

The largest model in the from-scratch ORCH lineup. Designed for full-stack Next.js generation: not just snippets, but complete project structures — TypeScript components, App Router pages, server actions, Prisma schemas, Tailwind utilities, and configuration files.

This is not a fine-tune of any pretrained model. Architecture and weights are trained end-to-end on curated Next.js repositories.

Architecture

Layers:                32
Hidden size:           2,560
Intermediate size:     10,240
Attention heads:       32
KV heads (GQA):        8
Max position:          16,384
RoPE theta:            10,000
Activation:            SwiGLU
Normalization:         RMSNorm
Tied embeddings:       no
Vocab size:            32,000 (custom)

Standard LLaMA-style ingredients (RoPE, GQA, SwiGLU, RMSNorm) at scale. The 16,384-token context length allows the model to keep a meaningful portion of a project in context during generation.

Training

• Data: curated Next.js repositories from GitHub
• Hardware: single NVIDIA A40 48GB (RunPod)
• Duration: ~2 hours
• Epochs: 3 (~29,000 steps)
• Batch size: 1 with gradient accumulation = 16
• Sequence length: 512 (training) — the 16K context length applies at inference
• Precision: BFloat16

Generated artifacts

The model is trained to produce complete project structures:

• Frontend: React components, App Router pages, layouts, hooks (TypeScript)
• Backend: API routes, server actions, middleware
• Database: Prisma schemas, query utilities
• Styling: Tailwind CSS, shadcn/ui-style component patterns
• Configuration: package.json, tsconfig.json, next.config.js

Usage

Custom PyTorch format — use the ORCH inference code (not yet published):

import torch
from orch.model.config import OrchConfig
from orch.model.transformer import OrchForCausalLM

model = OrchForCausalLM.from_pretrained("raihan-js/orch-nextjs-3b")
# ... use the same tokenizer.json from the repo

Intended use

• Full Next.js project bootstrapping from a natural language description
• Research into scaling SLMs trained from scratch on domain-specific data
• A from-scratch baseline to compare against fine-tuned models like ORCH-7B

Limitations

• Data-limited and undertrained: trained once on a rented A40 with a limited corpus (not compute-optimal), never benchmarked and never deployed. Treat output as research-grade, not production-grade.
• Training data scale: 3 epochs on curated Next.js repos. Don't expect the world knowledge of a 7B+ general-purpose model.
• Sequence length during training (512): capable of using long context at inference but may show degradation outside the training distribution.
• No safety alignment.
• Custom format: requires the ORCH inference code, not loadable with AutoModelForCausalLM.

Related models

• raihan-js/orch-fusion — 272M sibling, tiny 2,103 vocab
• raihan-js/orch-nextjs-350m-v2 — 287M sibling, 16k vocab
• raihan-js/orch-7b — alternate approach: QLoRA fine-tune of DeepSeek Coder 6.7B
• ORCH Studio — Gradio demo Space (currently runs ORCH-7B)

Author

Akteruzzaman Raihan Sikder — AI/ML engineer. Founding engineer and AI/ML lead at VETR Proposal; previously CTO of ClarioScope AI (2024–2026, sunset). Portfolio · GitHub.

Citation

@misc{sikder2025orchnextjs3b,
  title  = {ORCH Next.js 3B: A 3-Billion-Parameter Decoder-Only Transformer Trained From Scratch for Full-Stack Next.js Code Generation},
  author = {Sikder, Akteruzzaman Raihan},
  year   = {2025},
  url    = {https://huggingface.co/raihan-js/orch-nextjs-3b}
}