Cascade-1B-Logic
Public experimental byte-level causal language model. Training is in progress and checkpoints are uploaded continuously.
Architecture
- Approximately 1B stored parameters and approximately one third active per token.
- UTF-8 bytes, vocabulary 256, no external tokenizer.
- Four local causal Transformer layers process every byte.
- Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence. Over-cap boundaries are uniformly coalesced across the sequence instead of truncating its tail.
- Three independent eight-layer global routes are trained cyclically, one full batch per route.
- The previous completed global patch conditions the local prediction, preventing future-token leakage.
- Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
- BF16 training on one NVIDIA RTX PRO 6000 Blackwell, with batch size selected by a live memory/throughput probe.
Default context is 1024 bytes. This is a research checkpoint, not an instruction model.
Data mixture
The stream is not redistributed in this repository:
- 58% FineWeb-Edu sample-10BT
- 22% FineMath finemath-4plus
- 10% Cosmopedia v2
- 7% OpenR1-Math-220k
- 3% decontaminated Codeforces reasoning traces
Review each upstream dataset card and license before redistribution or commercial use.
Checkpoints
checkpoints/step-XXXXXXXX/model.safetensors contains BF16 model weights. training_state.json reports live loss, throughput, token count, batch size, routing and patch statistics.
Evidence and limits
The precursor 998.7M-parameter prototype reached validation byte loss 1.235 versus 1.360 for a 988.1M dense baseline on the same TinyStories test. It used about 4.15x fewer estimated model FLOPs and ran 2.06x faster wall-clock. This short controlled result motivated full training; it is not evidence of 100x speedup, broad reasoning ability or production readiness.
Training code and architecture are included for auditability. Results may change while training is active.