Built by VeriSafe.Live

Open-weights models, built, fine-tuned, quantized and served.

---

What VeriSafe.Live does

🚀 Now live at VeriSafe.Live: uncensored DeepSeek V4.1 Flash 1017, GLM 5.3 Flash and GLM 5.3. No wait time, instant access, crypto accepted.

---

Model highlights

• A non-transformer LM trained from scratch on a single M3 Max in stock MLX.
• 598M total parameters with ~78M active per token.
• Measured 10.5M-token context stream with constant 3.07 MB recurrent state.

---

Gala-598M — a Mac-first language model, trained from scratch on one M3 Max

Gala is a non-transformer LM designed around Apple Silicon's actual profile: lots of unified memory, modest FLOPs. Hoard parameters and state (cheap on a Mac), ration FLOPs per token (scarce). Everything here — pretraining, three-seed baselines, a 300M-token scale study, long-context finetunes, memory characterization, and a 10.5M-token context demo — ran on a single MacBook (M3 Max, 40-core GPU, 128 GB) in stock MLX. No CUDA, no custom kernels, no cluster.

Architecture

Four untied cells, each: Gated DeltaNet mixer (fixed-size fp32 fast-weight state) → hoard MLP (product-key-routed sparse blocks via mx.gather_mm: 1,024 blocks × 64 neurons, top-32 — a token touches 3% of a 537M-parameter pool) → 256-token sliding-window attention → hoard MLP. 598M total parameters, ~78M active per token. Tied embeddings. GPT-2 BPE.

Headline results (all measured, logs in results/)

Quality, matched data — 120M identical FineWeb-Edu tokens, same batch order, same optimizer, 3 seeds each:

Gap ≈ 10× seed noise. A tied-embeddings Transformer variant (3.860) also loses. At 300M tokens the lead grows: 3.581 vs 3.684.

Context scaling — 10,485,760 real FineWeb tokens streamed through this checkpoint, batch 1, on the laptop: ingest ~6 minutes, decode 382–388 tok/s flat at every depth, loss-on-next-2k flat from 32k to 10.5M, recurrent state 3.07 MB constant, peak RAM 5.3 GB. The matched Transformer decays 643 → 134 tok/s by 32k and would need ~33 GB of KV at 1M.

Memory, characterized honestly — passkey retrieval, 30 trials × 6 depths, chance = 20%: 100% (30/30) when the needle is inside the 256-token window, at both 8k and 32k context; chance at every depth beyond it. The state carries a measurable gist (loss at depth 2–4k into a document is 0.36 better than at depth 0–256) but not verbatim facts. The in-window control shows the boundary is architectural, not model capability.

Files

Use

pip install "mlx>=0.32" numpy tiktoken
python sample.py --run <dir-with-checkpoint-and-config> --prompt "The history of" --tokens 150
python bench_1m.py --run <dir> --total 10485760     # the 10M demo, ~7 min on an M3 Max

(To use these weights with the scripts: put model.safetensors + config.json in one directory — e.g. rename gala_u_300M.safetensors and gala_u_300M.config.json.)

Training

Muon (0.02, NorMuon-style row normalization) on hidden matrices + AdamW (3e-3) elsewhere, fp32 master weights and optimizer state, bf16 compute, fully-fp32 GDN mixer, mx.compile (+7.5%), seq 1024, batch 32,768 tokens, warmup 100 / 30% linear cooldown, grad clip 1.0. ~11.3k tok/s training throughput on the M3 Max.

One numerics note worth reading if you reimplement chunked delta rules (PAPER.md §6): computing the chunk's triangular inverse via the exact nilpotent identity (I−A)(I+A²)(I+A⁴)… detonates in fp32 once training correlates the keys (~500×-per-chunk state growth into inf, from finite inputs, through a provably bounded recurrence). Use blocked forward substitution. Instability class independently established by arXiv:2605.21325; the training-side account and the probe that found it (debug_nan.py) are in this repo.

Limitations

A research model, not an assistant: 420M training tokens total, FineWeb-Edu only, no instruction tuning, no downstream evals yet. Verbatim recall is window-local by construction (see above). Training wall-clock is ~1.5× slower than the matched Transformer on today's unfused MLX sparse path. Single machine, single dataset; scale conclusions beyond 598M/300M tokens are roadmap, not claims.

Citation

@misc{gala2026,
  title  = {Gala: a million tokens on a laptop, and what it cost},
  author = {VeriSafe.Live},
  year   = {2026},
  note   = {https://huggingface.co/junafinity/Gala-598M-MLX}
}

---

💬 Need a custom model?

If your company or research institute would like us to fine-tune any open-weights model, or train one from scratch, we'd be happy to help.

Drop us a line at info@verisafe.live

---

_{Built and published by VeriSafe.Live. Licensed under apache-2.0.}