Parallax 8B theta: mid-training checkpoints
These are research checkpoints from theta, a training run of Parallax. Training was stopped at step 6387 (156.2B tokens), before the planned end of the run; the final checkpoint is marked in the table below. Parallax is a mixture-of-experts language model trained on consumer GPUs spread over several countries. The hosts have no direct connections to each other and synchronize over the public internet.
- Base model. It is pretrained on web text only. It is not instruction-tuned, chat-tuned or safety-tuned, and it will continue text rather than follow instructions.
- Mid-training. Every export, the final one included, is a snapshot taken well before the planned ~973B tokens. Quality between exports is not monotonic.
- Research artifact. It is published so the training can be followed and inspected. It is not intended for production use.
Live training dashboard: parallax.chutes.ai.
Model
| Parameters | ~7.8B total, ~1.25B active per token |
| Layers | 64: 18 gated delta-rule (GDN2) recurrent, 8 sliding-window attention (window 2048), 6 sparse attention, 32 mixture-of-experts |
| Experts | 4096 routed experts (128 per MoE layer), 12 routed + 1 shared expert per token |
| Expert weights | ternary (-1, 0, +1) with per-row scales; at most two nonzero pairs in every group of eight |
| Width | d_model 1152; ReLU² expert activation |
| Tokenizer | Llama 3 tokenizer (vocabulary padded to 128,384); tied input/output embeddings |
| Training context | 4096 tokens |
Training
- Data: FineWeb-Edu, planned as a single pass over about 973B tokens. The published
validation number (
val_mix) is measured on held-out FineWeb-Edu files that are not in the training stream. - Optimization: the dense trunk is synchronized with a decoupled DiLoCo scheme over the internet. Each routed expert has one owner GPU that folds low-rank updates from the nodes that trained it into its full-precision master and publishes new ternary versions. The tech report describes the details.
- The learning rate was lowered by hand during the run, to 0.5x and later 0.25x of the planned peak, so later exports do not follow the planned schedule exactly.
Exports
One folder per export under exports/, named by training tokens (rounded down to whole billions) and
fleet step. Exports were taken about once an hour; training has stopped and no new exports will be added. Published exports are
never modified or removed.
| Export | Tokens | Step | Size | val_mix (nats/token) | 0-shot macro | 5-shot macro | Exported (UTC) |
|---|---|---|---|---|---|---|---|
156B-tokens_step6387 final (latest) |
156.2B | 6387 | 2.54 GB | 2.686 | 48.9 | 52.8 | 2026-09-28 13:50 |
151B-tokens_step6251 |
151.1B | 6251 | 2.54 GB | 2.682 | 48.9 | 54.0 | 2026-09-28 12:56 |
141B-tokens_step5944 |
141.3B | 5944 | 2.55 GB | 2.680 | 48.9 | 53.9 | 2026-09-28 11:55 |
131B-tokens_step5645 |
131.4B | 5645 | 2.56 GB | 2.685 | 50.0 | 54.2 | 2026-09-28 10:21 |
121B-tokens_step5351 |
121.6B | 5351 | 2.56 GB | 2.704 | 48.9 | 53.9 | 2026-09-28 09:21 |
111B-tokens_step5048 |
111.8B | 5048 | 2.56 GB | 2.757 | 49.3 | 52.7 | 2026-09-28 08:21 |
101B-tokens_step4744 |
101.9B | 4744 | 2.56 GB | 2.696 | 49.1 | 52.6 | 2026-09-28 07:21 |
92B-tokens_step4435 |
92.1B | 4435 | 2.55 GB | 2.704 | 48.7 | 52.4 | 2026-09-28 06:21 |
82B-tokens_step4129 |
82.2B | 4129 | 2.55 GB | 2.719 | 49.0 | 52.7 | 2026-09-28 05:21 |
72B-tokens_step3835 |
72.3B | 3835 | 2.56 GB | 2.745 | 48.3 | 51.5 | 2026-09-28 04:21 |
63B-tokens_step3356 |
63.5B | 3356 | 2.56 GB | 2.766 | 47.3 | 50.6 | 2026-09-28 03:21 |
55B-tokens_step2840 |
55.0B | 2840 | 2.56 GB | 2.809 | 46.5 | 50.0 | 2026-09-28 02:21 |
46B-tokens_step2289 |
46.5B | 2289 | 2.56 GB | 2.884 | 45.4 | 47.6 | 2026-09-28 01:20 |
38B-tokens_step1772 |
38.1B | 1772 | 2.56 GB | 3.002 | 44.6 | 47.0 | 2026-09-28 00:20 |
29B-tokens_step1249 |
29.5B | 1249 | 2.57 GB | 3.220 | 42.6 | 44.3 | 2026-09-27 23:20 |
19B-tokens_step674 |
20.0B | 674 | 2.58 GB | 3.815 | 38.2 | 40.2 | 2026-09-27 22:14 |
9B-tokens_step252 |
9.6B | 252 | 2.58 GB | 6.205 | 33.4 | 36.0 | 2026-09-27 21:09 |
- val_mix: mean cross-entropy (nats per token, lower is better) on the held-out FineWeb-Edu validation set.
- 0-shot macro: mean over 11 tasks (ARC-Challenge, ARC-Easy, BoolQ, COPA, HellaSwag, LAMBADA, OpenBookQA, PIQA, SciQ, SIQA, WinoGrande). 5-shot macro: the same tasks without LAMBADA (10 tasks). Per task, acc_norm is used for ARC, HellaSwag, OpenBookQA, PIQA and SciQ, and acc for the rest. Values are percentages.
- The scores come from the project's own scorer, which runs the native ternary experts. They track progress within this run. Compare them with numbers from other evaluation harnesses with care.
- A
-means the export has not been scored yet. The table fills in as scores arrive.
Latest export
exports/156B-tokens_step6387: 156.2B training tokens, step 6387.
hf download chutesai/parallax-8b-theta --include "exports/156B-tokens_step6387/*" --local-dir parallax-8b-theta
cd parallax-8b-theta/exports/156B-tokens_step6387
tar -xf packed_experts.tar
sha256sum -c --quiet SHA256SUMS # every file of the original export, byte for byte
Files and format
The files are in Parallax's native compact export format (inference only, no optimizer state). They are byte-identical to the export the training system produced:
| File | Contents |
|---|---|
manifest.json |
export manifest: tensor inventory, per-file sha256 digests, token clock |
model_config.json |
model configuration |
coverage.json, layouts.json |
tensor coverage and expert frame layouts |
indexer.bundle |
sparse-attention indexer weights |
relay_pack/ |
trunk (non-expert) weights in bf16, with their own manifest |
packed_experts.tar |
the 4096 routed experts (packed_experts/*.t24p, packed ternary codes and scales) |
SHA256SUMS |
sha256 of every file of the original export |
export_info.json |
step, tokens, time, sizes and digests of this export |
The only change from the original export is packaging. The 4096 expert files are stored in one
uncompressed tar to keep the repository's file count manageable. Extract it and check SHA256SUMS
as shown above. Every upload was checked against the training system's own digests before and
after it was published.
Running it. Standard transformers cannot load this format. A llama.cpp-based runtime for Parallax
and GGUF conversions of these checkpoints are in preparation (link to follow).
Tech report
The Parallax tech report: https://parallax.chutes.ai/tech-report.pdf. It is AI-generated from the project's measurements and logs, and it is a living document that changes as the run progresses. It describes the previous run (eta). Theta uses the same architecture with changes to the synchronization recipe.
Limitations
This is an early base model trained on educational web text. It can produce incorrect, biased or nonsensical output. It has no alignment or safety tuning, and it is small and far from converged.
License
MIT.
Table updated 2026-09-28 15:14 UTC.