Cubic Hier 150M
An experimental bilingual dialogue/reasoning language model with approximately 157M parameters. It keeps the faithful CubicV7 differentiable depth-memory path, uses hierarchical cosine retrieval in the two penultimate attention layers, and finishes with full global causal attention. Every sequence head uses QK RMSNorm.
Training recipe
- Base pretraining on a 72/13/10/5 stream of FineWeb-Edu, Russian FineWeb-2, deduplicated CodeParrot Python and FineMath-4+.
- Assistant-only SFT on Smol-SmolTalk and English/Russian Aya examples.
- Assistant-only reasoning SFT on the shortest verified-correct traces from OpenR1-Math-220k.
- DPO with an SFT anchor on Anthropic HH-RLHF and corrected UltraFeedback.
The optimizer is the project's hybrid orthogonalized Muon (large matrices and
a boosted depth group) plus AdamW for embeddings, norms, scalars and gates.
Base training uses warmup-stable-decay and a one-token-ahead MTP auxiliary loss;
embeddings, RMSNorm scales and controls are excluded from AdamW decay.
SFT teaches explicit direct and <think>...</think> system-prompt modes; set
CUBIC_REASONING=1 in chat mode to request the reasoning format.
Important
This is a custom PyTorch architecture, not a drop-in Transformers model. Run
train_and_chat.py with CUBIC_MODE=chat; it downloads/loads all required
files and starts an interactive console. The model is small and experimental:
verify factual, safety-critical and mathematical answers independently.
Each source dataset retains its own license/terms. FineWeb corpora inherit the Common Crawl terms described on their dataset cards; SmolTalk, Aya and OpenR1 are Apache-2.0; HH-RLHF and UltraFeedback use their published dataset licenses.