🎼 ORCH Next.js 350M v2 (287M)

Orchestrated Recursive Code Hierarchy — the second-generation 287M-parameter ORCH small model, trained from scratch with a Next.js-focused 16k-vocab tokenizer.

---

TL;DR

What this is

V2 of the small ORCH architecture. Same backbone as ORCH Fusion (24 layers, 1024 hidden, GQA 16/4) but trained with a larger 16k-vocab custom tokenizer focused on the Next.js / React / TypeScript ecosystem — common tokens for JSX, hooks, App Router file paths, Tailwind class names, and Prisma schema fragments compress efficiently.

Architecture

Model name:               orch
Size:                     350m (287M actual)
Layers:                   24
Hidden size:              1,024
Intermediate size:        2,816
Attention heads:          16
KV heads (GQA):           4
Max position:             4,096
RoPE theta:               10,000
Sliding window:           disabled
Activation:               SwiGLU
Normalization:            RMSNorm
Tied embeddings:          yes
Vocab size:               16,000

Training

• Data: synthetic code data emphasizing Next.js project structures
• Framework: custom PyTorch implementation
• Hardware: NVIDIA RTX 3060 12GB
• Precision: mixed
• From scratch — no warm-start from any pretrained model

Usage

Custom format — needs the ORCH inference code (not yet published):

import torch
from tokenizers import Tokenizer
from orch import OrchForCausalLM

model = OrchForCausalLM.from_pretrained("raihan-js/orch-nextjs-350m-v2")
tokenizer = Tokenizer.from_file("orch-tokenizer.json")

prompt = 'export default function HomePage() {'
ids = tokenizer.encode(prompt).ids
input_ids = torch.tensor([ids])
output = model.generate(input_ids, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0].tolist()))

Intended use

• Next.js / React / TypeScript code completion and small file generation
• Research baseline for SLM training on consumer hardware
• Comparing performance gains from vocab specialization (vs ORCH Fusion's 2,103-vocab)

Limitations

• Domain-specialized vocabulary: efficient on Next.js code, less efficient on Python, Rust, etc.
• Synthetic training data: see ORCH Fusion model card for caveats on synthetic vs real-world distribution
• No safety alignment

Related models

• raihan-js/orch-fusion — predecessor (272M, 2,103-vocab, project benchmark scores)
• raihan-js/orch-nextjs-3b — 3B sibling, custom 32k vocab
• raihan-js/orch-7b — QLoRA fine-tune of DeepSeek Coder 6.7B Instruct

Author

Akteruzzaman Raihan Sikder — AI/ML engineer. Founding engineer and AI/ML lead at VETR Proposal; previously CTO of ClarioScope AI (2024–2026, sunset). Trains SLMs from scratch and applies QLoRA fine-tuning on larger bases. Portfolio · GitHub.

Citation

@misc{sikder2025orchnextjs350m,
  title  = {ORCH Next.js 350M v2: A From-Scratch SLM with a Next.js-Focused Tokenizer},
  author = {Sikder, Akteruzzaman Raihan},
  year   = {2025},
  url    = {https://huggingface.co/raihan-js/orch-nextjs-350m-v2}
}