Parallax 8B kappa: mid-training checkpoints

These are research checkpoints from kappa, a training run of Parallax that is still in progress. Parallax is a mixture-of-experts language model trained on consumer GPUs spread over several countries. The hosts have no direct connections to each other and synchronize over the public internet.

• Base model. It is pretrained on a mixture of web, document, math, code, knowledge-focused and book text. It is not instruction-tuned, chat-tuned or safety-tuned, and it will continue text rather than follow instructions.
• Mid-training. Each export is a snapshot of a run that has not finished. Later exports are expected to differ, and quality between exports is not monotonic.
• Research artifact. It is published so the training can be followed and inspected. It is not intended for production use.

Live training dashboard: parallax.chutes.ai.

Model

Training

• Data: the "epsilon" data mix of an earlier Parallax run, not FineWeb-Edu. The main phase draws from seven sources totalling about 1.09T tokens: two general pretraining mixtures of web, document, math and code text (about 78% of the tokens), an additional web set, two math sets, a knowledge-focused set and a books set. Two annealing sources join near the end. The run is planned as one pass of about 973B tokens.
• Optimization: the dense trunk is synchronized with a decoupled DiLoCo scheme over the internet. Routed experts are trained through low-rank adapters on the GPUs that use them, and the adapter updates are folded into full-precision masters that publish new ternary versions. The tech report describes the details.
• Learning rate: peak 4e-4 after an 8.7B-token warm-up, then constant, with a planned 1-sqrt decay to 0.1x of the peak between about 827B and 973B tokens. After gradient-noise instability, the rate was cut during the run to 0.5x (2e-4) on 2026-09-30 at 07:05 UTC and to 0.25x (1e-4) at 13:00 UTC.
• Batch: 10 gradient-accumulation micro-batches per step (about 8.2M tokens per fleet step) from the start, raised to 20 (about 16.4M tokens per step) on 2026-09-29 at 06:26 UTC. It ran at 40 from about 08:00 to 14:42 UTC on 2026-09-30, then returned to 20.
• Output logits: the learnable output logit scale is capped at 3.0 in training and inference (see Files and format).

Exports

One folder per export under exports/, named by training tokens (rounded down to whole billions) and fleet step. A new export is added about once an hour while the run continues. Published exports are never modified or removed.

• val_mix: mean cross-entropy (nats per token, lower is better) on a fixed held-out validation set stratified over the nine sources of the epsilon mix (2048 windows of 4096 tokens). It is not comparable with the FineWeb-Edu validation number published for theta.
• 0-shot macro: mean over 11 tasks (ARC-Challenge, ARC-Easy, BoolQ, COPA, HellaSwag, LAMBADA, OpenBookQA, PIQA, SciQ, SIQA, WinoGrande). 5-shot macro: the same tasks without LAMBADA (10 tasks). Per task, acc_norm is used for ARC, HellaSwag, OpenBookQA, PIQA and SciQ, and acc for the rest. Values are percentages.
• The scores come from the project's own scorer, which runs the native ternary experts. They track progress within this run. Compare them with numbers from other evaluation harnesses with care.
• Benchmark prompts are scored with every run of two or more newlines collapsed to a single newline (including the few-shot separator).
• A - means the export has not been scored yet. The table fills in as scores arrive.

Latest export

exports/881B-tokens_step60735: 881.5B training tokens, step 60735.

hf download chutesai/parallax-8b-kappa --include "exports/881B-tokens_step60735/*" --local-dir parallax-8b-kappa
cd parallax-8b-kappa/exports/881B-tokens_step60735
tar -xf packed_experts.tar
sha256sum -c --quiet SHA256SUMS   # every file of the original export, byte for byte

Files and format

The files are in Parallax's native compact export format (inference only, no optimizer state). They are byte-identical to the export the training system produced:

The only change from the original export is packaging. The 4096 expert files are stored in one uncompressed tar to keep the repository's file count manageable. Extract it and check SHA256SUMS as shown above. Every upload was checked against the training system's own digests before and after it was published.

Logit-scale bound. The model's output logit scale is bounded: the forward pass uses exp(min(logit_scale_log, log 3.0)). The stored trunk tensor is the raw training parameter (it can sit slightly above the bound, e.g. from bf16 rounding), and each export records the bound in manifest.json (logit_scale: max, raw, effective) and in coverage.json (inference_policy.logit_scale_max). A loader must apply the recorded bound; the files themselves are left byte-identical.

Running it

This is a base model only. It is not chat- or instruction-tuned, so it does plain text completion: give it the start of a text and it continues it. It will not follow instructions or hold a conversation.

Standard transformers cannot load this format. Use our llama.cpp fork, https://github.com/chutesai/llama.cpp, which adds a dedicated runtime, llama-parallax, for CPU (x86-64, ARM64) and Apple GPUs (Metal):

git clone https://github.com/chutesai/llama.cpp && cd llama.cpp
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON   # -DGGML_METAL=OFF for CPU only
cmake --build build --target llama-parallax -j 8
python tools/parallax/run.py --binary build/bin/llama-parallax --model parallax-t9.gguf \
  --tokenizer tokenizer.json --backend metal --experts lut9 \
  --prompt 'The capital of France is' --predict 64

The runtime reads GGUF files converted from these exports. A ready-made GGUF is published at the repository root: parallax-8b-kappa-702B-tokens.gguf (2.21 GB, sha256 9eec18c9ff094c272ac789cbdf88e89a15f6381f0f77616f51d9486a2613bef2), converted from exports/702B-tokens_step48358 (702.7B training tokens, step 48358) with a bf16 trunk and the exact ternary experts. It is a one-off snapshot and is not updated with new exports; convert a newer export yourself for a later checkpoint. Earlier snapshots, still in the repository: parallax-8b-kappa-627B-tokens.gguf (from exports/627B-tokens_step43524), parallax-8b-kappa-612B-tokens.gguf (from exports/612B-tokens_step42498), parallax-8b-kappa-568B-tokens.gguf (from exports/568B-tokens_step39678), parallax-8b-kappa-528B-tokens.gguf (from exports/528B-tokens_step37110), parallax-8b-kappa-408B-tokens.gguf (from exports/408B-tokens_step29319), parallax-8b-kappa-368B-tokens.gguf (from exports/368B-tokens_step26688).

hf download chutesai/parallax-8b-kappa parallax-8b-kappa-702B-tokens.gguf --local-dir .

Use it as --model parallax-8b-kappa-702B-tokens.gguf in the command above. tools/parallax/README.md in the fork covers conversion, options and the tokenizer.

Tech report

The tech report for this run (kappa) is archived in this repository as tech_report.pdf. It is the version served on the project site from 2026-10-02 and is AI-generated from the project's measurements and logs. The project's current tech report is a living document and may describe a later run.

Limitations

This is an early base model. It can produce incorrect, biased or nonsensical output. It has no alignment or safety tuning, and it is small and far from converged.

License

MIT.

_{Table updated 2026-10-03 10:43 UTC.}