Positronic v1

Model will be ungated for open download soon! These models are undertrained, and are NOT meant to be finished. These are Research Artifacts only.

144M causal LM with a non-standard channel mixer. Token mixing is still attention. Per-layer MLP is a QuazimotoBlock: coupled phase oscillators (Kuramoto) on concentric rings, a few differentiable Euler steps, readout [cos θ, sin θ].

Positronic is from Data's positronic brain in Star Trek - a mind on weird internal dynamics, not a standard FFN stack.

Same tokenizer, traits, and data as the sibling Quazimoto models (MyCel-LM). Only the mixer is swapped. Controlled experiment: can a transformer generalize if the FFN is a self-organizing dynamical system instead?

Research / hobby architecture, not a product. 144M, short run. Fluent ChatML, can reproduce <think>…</think> structure. Facts and arithmetic are unreliable. Interesting for what it is, not what it knows.

• Base checkpoint: pretraining, ~64k steps (base weights not shipped in this repo)
• This release (SFT): chkpt/quazimoto_sft.pt (chat/instruction SFT, 8k steps)
• Params: 144.3M
• Vocab: 16,512 (byte-level length-max tokenizer + appended special tokens)
• Context: block_size 2024 (RoPE max_position_embeddings 4096)

---

Architecture at a glance

Every layer is x → x + Attention(norm(x)) then x → x + QuazimotoBlock(x). All the exotic machinery lives in (or around) the QuazimotoBlock.

1. QuazimotoBlock - the oscillator channel mixer

This is the defining component. Instead of an MLP, each token's hidden state drives a bank of N = 508 phase oscillators laid out on 7 concentric rings:

1. Project to phases and frequencies. Two linear maps produce initial phases θ [B,T,N] and bounded natural frequencies ω = tanh(...).
2. Structured Kuramoto coupling. The oscillators are coupled through a learnable R×R (7×7) block-gain matrix between rings - strong on-diagonal (within-ring) coupling, weaker neighbour-ring coupling at init. Because coupling is block-constant, the update uses per-ring mean fields (order parameters), giving an O(N·R) update instead of the O(N²) pairwise Kuramoto sum. A per-ring learnable frustration α shifts the coupling phase.
3. A rotating "center ball." A single global phase (center_phase, advancing at center_omega) pulls every oscillator toward it via a per-ring center_gain - a hierarchy-to-center coupling that is the "Quazimoto" twist on plain oscillatory neurons.
4. Integrate. osc_steps=4 differentiable Euler steps advance θ under ω + coupling + center-pull + external drive.
5. Read out. Concatenate [cos θ, sin θ] → [B,T,2N], pass through a 3×-expansion GELU MLP back to d_model.
6. Family gate + soft-clamp. Output is scaled by tanh(gate) and passed through soft_clamp (an erf saturating clamp, bound 10) for BPTT stability. Unlike the auxiliary traits below, this gate is initialised open (~0.9) because the oscillator bank is the main mixer.

Degenerate check: set ring coupling to dense and α=0 and the block reduces toward AKOrN-style oscillatory neurons. The ring structure + center coupling is what makes it Quazimoto.

2. Attention - MLA / partial-RoPE / GQA

Ported from a "v2" transformer. Multi-head Latent Attention (low-rank q/o projections, rank 384), partial RoPE (each 64-dim head splits into a 32-dim no-position part and a 32-dim RoPE part), QK-Norm (per-head RMSNorm before RoPE), and GQA (4 KV heads shared across 12 query heads). Optional DERF (erf-attention) and XSA (value-subspace removal) exist but are off in this checkpoint. A GQA-compact KV cache supports incremental decoding.

3. Interstitial rings (use_rings=True)

Between each pair of adjacent oscillator rings sits a slot that absorbs context once and injects a constant drive into its two neighbour rings during the Kuramoto integration:

• PhaseAttentionRing - causal self-attention in phase space ([cos,sin] of the two neighbour rings) producing an injection current.
• EngramRing - an n-gram hash memory: a frozen LSH-style compressor hashes 1-3-token windows into per-head embedding tables, gated by a DERF context gate, projected into an injection current.

4. Per-ring self-organizing controllers (use_ring_controllers=True)

One tiny RingController per ring (shared across layers). Each observes its ring's order-parameter magnitude/phase and mean frequency, and emits no-op-at-init modulations of that ring's coupling, frustration, center-pull, and injection. Its core is a fast-weight predictor updated online by a delta rule (one gradient step on its own prediction error - surprise minimization), not by backprop; only a small zero-init decoder is backprop-trained.

5. Per-ring memory specialists (use_ring_specialists=True)

A small MoE of 7 mini "memory specialists" per ring (top-2 routed). Each specialist owns two test-time-writable stores (store_in, store_out) that accumulate an addressable context memory during generation via a gated EMA write. Tokens route to specialists by similarity to a learnable identity key plus a read of the accumulated input memory; the routed store_out is decoded into a ring injection current.

Important for inference: these stores are mutated at inference time and persist across forward calls. See "Known issue" below.

6. Trunk refinements & auxiliary heads

Applied after the layer stack, before the final RMSNorm + LM head:

• HRM (use_hrm=True) - Hierarchical Reasoning Module: iterative gated refinement that starts from a random learned state z0 and reconciles it against the trunk over 3 steps, contributing only the reasoning delta. Gates start open.
• MoE SwiGLU (use_moe=True) - shared + top-2-of-4 routed SwiGLU experts with load-balance aux loss; down-projections zero-init (no-op at start).
• MTP (use_mtp=True) - 4 multi-token-prediction draft heads. A train-time auxiliary loss, and at inference they enable self-speculative decoding (--speculative): the heads draft the next 4 tokens, the main head verifies them in one parallel forward. Output is bit-identical to greedy decoding, only faster.
• JEPA (use_jepa=True) - representation-space k-ahead prediction with a stop-grad target. Train-time only (auxiliary loss); no effect at inference.

7. Fractal (Mandelbrot) phase seeding (use_fractal_phase_seed=True)

Each token id maps to a complex point c (spread over the Mandelbrot region by a 2D Halton sequence); the angles of its z ← z² + c orbit seed the oscillator phases, added through a zero-init gate. A frozen, parameter-free, token-specific dynamical prior congruent with what the Kuramoto block already does. The [16512, 508] table is regenerated deterministically on load (not stored in the checkpoint).

Family "safe-at-init" contract: every auxiliary block (rings, engram, specialists, HRM, MoE, MTP, JEPA, fractal seed) is gated so it is a no-op at initialization while its content weights still receive gradient - the model boots as a clean attention+oscillator transformer and the optimizer opens each trait only if it helps.

---

Verification (this checkpoint)

I confirmed the released SFT checkpoint actually uses the full custom architecture (not a fallback):

• Strict state_dict load succeeds - every checkpoint tensor maps onto the model with none missing or unexpected. (generate.py loads with strict=False, but strict passes too.)
• All traits are active in family_config: HRM, MoE, MTP, JEPA, Rings, Controllers, Specialists(7/ring), FractalSeed.
• Training opened the oscillator block: per-layer output gates tanh(gate) climb from ~0.31 (layer 0) to ~0.99 (deep layers) - the oscillators dominate the later layers.
• Ablations change real-prompt logits (max |Δlogit| on one assistant-turn prompt): zeroing the oscillator gates 31.3 (argmax flips), MoE 12.7, ring controllers 8.1, HRM 3.2, fractal seed 0.63, interstitial rings 0.12, ring specialists 0.06. So the big mixers are load-bearing; the fractal seed and interstitial rings are present but contribute weakly at this checkpoint.
• KV-cache path is correct: greedy decoding with use_cache=True is bit-identical to full recompute over 40 tokens.

---

What this is for / what it isn't

Research into oscillatory / Kuramoto channel mixing, custom-architecture inference, small ChatML/<think> demo.

Not for factual QA, arithmetic, or anything that needs to be correct. 144M, short run. Format is often right; content is often wrong or incoherent.

No safety / RLHF. Outputs can be nonsense. English-centric.

---

Usage

transformers note

Custom architecture (QuazimotoLM in model.py). Does not load via AutoModelForCausalLM. Use this repo's model.py + generate.py. Tokenizer is HF-compatible via SpikeTokenizer (spike_tokenizer.py).

Chat generation (recommended for the SFT model)

python generate.py --ckpt chkpt/quazimoto_sft.pt --chat \
    --prompt "Hello, who are you?" --max_new_tokens 200

Chat mode wraps the prompt in ChatML (<|im_start|><|user|> … <|im_end|> then an <|assistant|> header) and stops on <|im_end|>.

Reasoning with <think>

The SFT mix included short-reasoning data, so the model knows the <think>…</think> format but does not always open the tag on its own. To force it, prime the assistant turn ending in <think>.

Speculative decoding (faster, greedy-identical)

python generate.py --ckpt chkpt/quazimoto_sft.pt --chat --speculative --spec_stats \
    --prompt "Explain the plan." --max_new_tokens 200

Programmatic

import torch
from model import QuazimotoLM, QuazimotoConfig
from generate import load_tokenizer, generate, build_prompt, resolve_stop_ids

tok = load_tokenizer(".")
ck = torch.load("chkpt/quazimoto_sft.pt", map_location="cpu", weights_only=False)
model = QuazimotoLM(QuazimotoConfig(**ck["family_config"]))
model.load_state_dict(ck["model"]); model.eval()

model.reset_ring_memory()               # clear test-time specialist memory per request
ids, _ = build_prompt(tok, "Hello!", chat=True, system="")
x = torch.tensor([ids])
out = generate(model, model.cfg, x, 200, stop_ids=resolve_stop_ids(tok, True))
print(tok.decode(out[0, len(ids):].tolist(), skip_special_tokens=True))

Special tokens

ChatML (<|im_start|>, <|im_end|>, <|system|>, <|user|>, <|assistant|>), reasoning (<think>, </think>, <begin_solution>, <end_solution>), tool calls (<tool_call>, <tool_response>), code FIM (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>), and <|endoftext|>.

---

Statefulness note (ring-specialist test-time memory)

The ring specialists write to persistent stores that carry over across forward calls (store_out norm goes 0.0 → ~44.5 after one forward), so in principle memory from one prompt can leak into the next. At this checkpoint the practical effect is negligible: the specialist output gates barely opened during the short SFT (tanh(scale) in -0.10…+0.17), so cross-prompt contamination perturbs next-token logits by only ~0.007 (max) and did not flip a single greedy token in testing. It is the weakest of all the model's components.

It is still handled correctly for reproducibility and to future-proof checkpoints where those gates open wider: generate.py now calls model.reset_ring_memory() before every prompt (REPL turn / single run). Flags:

• --keep_ring_memory - opt out of the per-prompt reset (let memory accumulate across turns).
• --freeze_ring_memory - disable online writes entirely for fully stateless, reproducible generation (model.set_ring_memory_writing(False)).

A Gradio wrapper that reuses one loaded model gets clean per-request behavior through the same run_once path; if you call generate() directly, call model.reset_ring_memory() yourself at the start of each request.

---

Training data

• Pretraining blend: ~35% Ultra-FineWeb-L3, 25% FineWeb-Edu, 25% FineMath, 15% Quazim0t0/PretrainNew.
• SFT blend (this release): ~equal quarters of mlabonne/ultrachat_200k_sft, mlabonne/ultrafeedback-sft, openbmb/UltraData-SFT-2605 (Knowledge, no-think split), and volcanos/OpenThoughts2-1M-ShortThink (reasoning). Loss is masked to assistant turns (standard instruction SFT).

Files

model.py (architecture), family.py (trait modules), fractal.py (Mandelbrot seeding), generate.py (inference harness), spike_tokenizer.py + tokenizer.json (tokenizer), special_tokens.py (special-token registry), chkpt/quazimoto_sft.pt (this release).

Citation

If you use this model, please cite:

@misc{positronic144m,
  title        = {Positronic-144M: A ~144M-parameter Quazimoto oscillator LM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Positronic-144M}},
  note         = {Quazim0t0/Positronic-144M}
}

Update: format-blended SFT

Short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train ('#### N' reasoning), and MMLU-style ('Answer: ') examples, so chat fluency improves while the benchmark output-formats are preserved. Held-out (test-split) before->after:

held-out 1.96->1.18, MMLU acc 0.040->0.255, format 0.225->0.995 (A/B test showed opening the inert EngramRings adds nothing over plain SFT, so this ships with rings unchanged.)

Note: fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.

Update: longer-trained checkpoint + format-blended SFT

Same 144.3M-param architecture, trained further (to step 8250, ~1B tokens, with the context window extended 2048 -> 4096), then given the format-blended SFT (smoltalk + GSM8K '#### N' + MMLU 'Answer: '). Held-out (test-split) before->after: held-out loss 2.01 -> 1.12, MMLU acc 0.010 -> 0.255, answer-format rate 0.095 -> 1.000.

Best decoding (from a decode_lab.py sweep on this checkpoint): temperature ~0.7, top-k 40, repetition-penalty 1.15, no-repeat-ngram 3 (the "nrng" recipe; distinct-2 0.945 / rep-4 0.000). Do NOT use CFG on this model -- it over-amplifies and degenerates ("first first first ..."). Greedy + no-repeat-ngram is also stable.