Chat Preview

Byrne-100M-Ultra-MC

October 11, 2026 update: Memory Cache recall test

I ran a long-range recall test on the base checkpoint (base_62k.pt, step 62,000) to measure what Memory Cache (MC) adds. A 128-token wikitext-103 passage is repeated after D tokens of other text. The score is how much easier the second copy is to predict than the same passage without a first copy (nats, higher is better). Context 1024, 8 samples per distance.

• MC improves recall at every distance, and the gain grows with distance, from +4% at 64 tokens to +16% at 760. Attention handles recent text; MC carries more of the older context.
• MC also lowers ordinary text cross-entropy by 0.099 nats (about 4%), so it is used for language modelling in general, not only for repeats.
• The trained gates are already at their best setting: widening them x1.25 or x1.5 adds nothing at long range and makes text CE slightly worse.

The long-range recall effect was also confirmed on On-Fly-Jev-v2 (the Fly / BioWhale-Fly100 trunk with Memory Caching). With MC off, its recall at 1,400-1,760 tokens drops to about zero (0.002-0.004), and with MC on it is 0.057-0.059, so MC provides its long-range recall. That test uses a 2,048-token context, so its numbers are not directly comparable to the table above.

Reproduce: python recall_mc_byrne.py checkpoints/base_62k.pt (raw numbers in mc_recall_results_step62000.json). This is a measurement only; no weights changed.

September 12, 2026 update

I went back through the Memory Cache and attention code and fixed several execution bugs. These are code updates. I have not retrained or replaced the checkpoint weights for this update.

• MC now keeps its memory when decoding continues with several tokens at once.
• Masked tokens are excluded from MC memory updates and segment summaries.
• Explicit attention masks now retain the causal boundary. Previously, supplying a normal binary padding mask could let attention see future tokens.
• Binary padding masks and standard additive causal masks now work together across attention and MC. Arbitrary query-specific masks are rejected by MC.
• Empty supervised targets no longer produce NaN cross-entropy, including MTP.
• Training no longer builds unused MC inference caches. Short prefixes also avoid padding the branch's computation to a full 256-token segment.

The checks passed on CPU: causal masking, masked chunk continuation, finite gradients, and the DPO checkpoint's four-token cached/full-recompute comparison. All 503 checkpoint tensors still load. A 16-token MC branch test measured 2.20 ms before and 0.64 ms after the short-prefix change. That is a branch microbenchmark, not a 3.4x speedup for the whole model. Small floating-point differences are possible because the matrix shapes changed.

I also checked generations. With the same seeds, atomic role template, and sampling settings, all 14 responses stayed identical after the efficiency changes. I then raised the output allowance from 96 to 512 tokens. Some answers continued much further, but the extra text did not fix the answers. The bus question reached 29 passengers in its reasoning instead of 44 and continued with inconsistent calculations. Some responses still hit the larger limit. I am not claiming these fixes improved instruction following or retrieval.

The generation tests use the original SFT/DPO role tokens: <|im_start|><|user|>\n...<|im_end|>\n<|im_start|><|assistant|>\n. The shipped chat helpers still use literal user and assistant role names; that template discrepancy is separate from the MC changes. Use the explicit test command below to reproduce these results.

python test_daisy_smoltalk_sft_generations.py --ckpt checkpoints/dpo_3200.pt --out generations_512.json --code-dir . --atomic-roles --max-new-tokens 512

Updated code: model_v2.py and generation test. Checks: MC audit and architecture audit. Results: architecture checks, MC checks, branch timing, 96-token generations, and 512-token generations.

The automatic generation scorer only checks a few keywords and can credit wrong answers. Treat the saved response text as the evidence, not those pass counts. Also, an earlier explanation that SFT loss was reported four times too low was incorrect: summing the accumulation-scaled microbatch losses gives their average. The saved field called val_loss was a training value, however.

114M looped transformer (SpikeWhale v2 / Byrne) with a parallel Memory Cache branch. Trained from scratch on a web/code/math mix, then SFT, then DPO. All three stages are in this repo.

It's a small model. Fluent English, simple chat formatting, not a knowledge base, weak at code and multi-step reasoning. Numbers in Evaluation. morpho/ is a side project: the same block grown as an int8 gate circuit. It is not on the generate path.

The base was trained on 2.0B tokens. SFT adds 0.23B, DPO adds 0.05B. 2.3B in total. Almost all of that is the base.

A smaller looped sibling lives in Byrne-15M-Looped (~15.2M active, loop_count=3). That was the toy / smoke-test for this run: looping at 15M before I spent the 114M budget. Not a matched scale-up.

Where to look

Checkpoints

Same architecture family, same loader for the 114M stages. The 5M nets are a separate, smaller stack. Weights and context length differ: SFT was trained at 4096, base and DPO at 1024.

Architecture

Looped decoder (model_v2.py, SpikeWhale v2 / Byrne). Dense — use_moe is off; FFN width is still moe_intermediate_size 2048. Vocab 16,512 via a byte-level Length-MAX tokenizer (spike_tokenizer.py, tokenizer.json) from Dong & Su, Length-MAX Tokenizer for Language Models (arXiv:2511.20849, 25 Nov 2025). Text → UTF-8 → latin-1 bytes, then greedy longest-match against the vocab. Not a HuggingFace tokenizers file; the wrapper is what AutoTokenizer talks to.

Config lives in config.py + byrne_100m_ultrax_mc.yaml. Released .pt files carry the config they were trained with. Engine and tools read that, not the yaml, because the stages disagree about context length.

Loop

The 16-layer stack runs three times with the same weights. Effective depth 48, parameter cost of 16 layers plus a tiny per-pass embedding (loop_pass_embed, zero-init). loop_count=1 is the dense baseline.

loop_layer_plan walks (layer, pass) pairs. Cache slots are the plan index, not the layer index. Same weights, different activations each pass, so pass 1's K/V cannot share a slot with pass 0. Depth-attention state is reset at the start of every pass so pass 3 does not attend to a stale pass-1 value of the same layer.

loop_mode can be "full" (this model) or "middle_split" (unlooped prefix/suffix, only the middle repeats). Not used here. The engine still sizes the KV cache from the plan, so that mode would not silently break.

MLA, XSA, QK-norm

Multi-head latent attention: Q through q_lora_rank 128, output through o_lora_rank 128, RoPE only on qk_rope_head_dim 16 of each 64-dim head. 2 KV heads (GQA). Per-head RMSNorm on Q and K before RoPE (use_qk_norm). z-loss 1e-4 on the LM head.

XSA (exclusive self-attention) is on: an orthogonality correction that pulls the self-echo out of the attention output. DERF and Elo attention exist in config.py and are off.

Depth attention

Cross-layer, not cross-token. At layer L the current V is mixed with V from earlier checkpointed layers (stride 4), weights from a softmax over the depth axis. Learned skip along depth. One gate parameter per layer, zero-init, no-op at step 0. The attention forward takes depth_kv and returns depth_entry. A loader written for the Mark2 signature misses that and falls back to torch.cat cache growth.

Hyper-Connections, Engram, HRM, MTP

• Hyper-Connections (hc_mult 2): two residual streams with learned routing between layers. Final mix is learned (hc_out_mix, init = mean).
• Engram: hash-table n-gram memory into the embeddings. Compress dim 32, 2 heads, table 4096, max n-gram 3. This tree fixed lookup internally (EngramModule.lookup(prefix)); there is no engram_context_ids kwarg. The engine threads the prefix through a trailing cache slot.
• HRM refinement: one small block after the stack, 1 inner step, dim 128, deep supervision on during training. Inference reads the last step only. Gate init is nonzero so the zero-init up still gets a gradient.
• MTP: one extra head, loss weight 0.3. At inference the engine uses it as a speculative draft. Identity-checked against the main head.

Memory Cache

Adapted from Behrouz, Li, Deng, Zhong, Razaviyayn & Mirrokni, Memory Caching: RNNs with Growing Memory (Google Research, arXiv:2602.24281, 27 Feb 2026). Runs next to MLA as a gated residual.

The paper's setup, short version: transformers keep every past token addressable and pay O(L²) for it. RNNs squash history into a fixed state at O(L) and forget. They split the sequence into segments, stash a checkpoint of the recurrent memory at each boundary, and let a query read the live state plus those checkpoints. Segment count slides you between the two ends.

Here it sits per layer beside softmax attention: mc_num_heads 4, mc_head_dim 32, mc_gate_dim 64, mc_segment_len 256.

1. Cut the sequence into segments of 256.
2. Per segment i, Katharopoulos map φ(x) = elu(x) + 1, accumulate M⁽ⁱ⁾ = Σₘ φ(kₘ) vₘᵀ and z⁽ⁱ⁾ = Σₘ φ(kₘ). Fixed-size summary of the segment, however many tokens went in.
3. Token t in segment s reads φ(q_t) M⁽ⁱ⁾ from every i ≤ s. Own segment is a causal running memory (tokens up to t only). Earlier segments are finished, so their full memory is used as-is.
4. Gate: γ_{t,i} = softmax_i ⟨u_t, meanpool(x over segment i)⟩, masked to i ≤ s. Per token, per layer, pick which segment to look at.
5. One global normaliser, numerators left unnormalised: y = Σᵢ γ·num / (Σᵢ γ·den + ε).
6. Mix in through a zero-init gate: attn_out = attn_out + tanh(mc_gate) · mc_out.

Two bugs that had to stay fixed:

• Diagonal gate needs a causal running mean. Mean-pooling the whole current segment leaks future tokens into token t. Implementation is a cumsum running mean on the diagonal.
• Per-segment normalisation kills it. You lose query–key match magnitude and the only thing left distinguishing segments is the coarse mean-pool gate. Global normaliser is what keeps match strength in the signal.

Runs in fp32, autocast off. The linear-attention denominator underflows in bf16 and training blows up.

This is not an RNN, so the paper's complexity story does not transfer cleanly. Cross-segment readout here is O(N²L). It is not a long-context speed trick. The paper's "growing memory" is with sequence length (more segments → more cached states), not with training steps. What grows with training here is the zero-init mix-in tanh(mc_gate): at step 0 the model is the no-MC baseline; the residual only starts mattering if that gate opens. On this 114M run it did (knock-out ~9% early → −16.7% at DPO; mean |tanh| ~0.60). Pretrain seq_len was 1024 = 4 segments, so the branch sees cross-segment memory from the first step.

It did get used. DPO 3.2k checkpoint:

Gates are open. Knocking the branch out costs ~17% PPL (it was ~9% earlier, before they opened). That is an inference knock-out on this checkpoint, not a model trained without MC. The 114M twin also dropped fractal RoPE, so it is not an MC-only ablation. A 5.7M 2×2 that does isolate the knobs is in 5m-ablation/. MC alone did not help there. MC+fractal together beat std at train length on seed 1 and did not on seed 2.

If you write a loader: this branch is a full-sequence op. It uses the whole x given to forward. Naive KV cache feeds it one token, it rebuilds segments from a 1-token window, and you get a different function than training. Still fluent. Gate here is ~0.56–0.60 (Mark2 siblings sit around 0.19), so the damage is worse: max |Δlogit| 7.24e-01 vs a full recompute, against a ~1e-5 float32 floor. Shipped code keeps incremental state per segment and per loop pass (loop_count=3, same branch three times on the same positions). That is what makes cached decode match a full recompute. python verify.py (generate.py / model_v2) or python verify_cache.py (spike_infer).

Fractal RoPE

Cantor-spectrum RoPE (gamma=1) instead of the usual geometric schedule. Same endpoints and band count, exponents on the middle-thirds Cantor set. See fractal.py.

In-distribution (this 62k net). Swap the trained model to geometric RoPE at inference (same weights, different frequencies) and natural-text PPL blows up. That is dependence, not “fractal is better”:

Both models, native vs swapped frequency table. Same weights, only inv_freq changes (eval_rope_swap.py). Ultra-MC 62k trained fractal → geometric at load. Twin 38.5k trained geometric → fractal gamma=1 at load. Unique non-repeating records, 3 windows. A blowup means that schedule is load-bearing for that net.

PPL by target position on a 2048 window (same probe):

What that actually says:

• Inside train length, Ultra-MC is tuned to fractal frequencies. Swap to geometric is already wrong by 512, and 512–1024 is ~50–100×.
• Past 1024 this fractal-trained 62k net dies even on its own schedule (first bad bin 1024–1280, then gone). Geometric swap is also dead out there. So fractal is load-bearing in 1024 and does not carry this probe to 2×.
• The twin at 38.5k, geometric, holds through 1536 and only blows the last 256 tokens (112 PPL). Put fractal frequencies on those weights and the last bin drops to 3.06; whole-seq 2048 goes 4.48 → 2.76. In 1024 the swap is a small hit. So geometric is load-bearing where it was trained, and Cantor frequencies at inference are what flatten the twin past 1024 on this probe.
• Twin 38.5k in this table is a mid-run snapshot. Finished 61k numbers (and the 2048 last bins) are under Matched 114M runs. Do not read “fractal is worse at 2048.” This 62k fractal net does not extrapolate here; the geometric twin does.

What I can and cannot claim about Memory Cache

Behrouz et al. 2026 (arXiv:2602.24281) score MC with matched RNN trains, then Wiki PPL, S-NIAH (passkey / numeric / UUID), in-context recall vs length, and MQAR. Those are the tests I ran on this 62k base, MC on and with mc_gate zeroed.

They changed the direction. The earlier filler/needle story was not the paper’s recall eval. On S-NIAH and MQAR this 114M base is at chance whether the branch is on or off. You cannot prove MC helps recall if the model cannot do the task either way.

The 1-hit gaps are n=20 and n=6 cells. That is noise, not a result.

There is no 114M test that reliably proves Memory Cache does or does not help. Unsure at this size, and stated as such. The 5.7M 2×2 below is the isolation run; read that as 5M, not as this card.

How it may be helping, if it is (this is not proof):

• Gates are open. DPO 3.2k mean |tanh(gate)| = 0.599 across 16 layers (range −0.75 … +0.90). Training used the residual.
• Inference knock-out on this checkpoint: UltraX PPL 9.87 → 11.86 (−16.7%) when mc_out is removed. Earlier in training the gap was ~9%. That only says the branch is in the next-token CE path here, not that a twin trained without it would be worse.
• Pretrain seq_len 1024 = 4 × mc_segment_len 256, so the branch saw more than one segment from step 0. That is the setup, not a score.

A matched train that changes only MC at 114M (fractal left on) still does not exist. The 114M twin dropped both knobs. A 5.7M 2×2 on the current FineWeb-Edu blend, 10k steps, same seed recipe, does exist. That is the isolation test. It is not this 114M card.

5.7M matched 2×2 (10k steps)

Same blend as the nomc twin. Four runs on seed 1. Seed 2 retrained std and both only. Only use_memory_cache and use_fractal_rope change. Eval is deterministic (repeat matched to the printed digits). The seed gap is training, not measurement.

WikiText-2 PPL @1024 (20 windows):

Unique-text PPL (native):

Seed 1 val loss at 10k: both 4.87, std 4.96, fractal 5.00, mc 5.06.

MC alone does not help at this size. Worse PPL, worse stretch (seed 1; not retrained). Fractal alone does not help. Same. Swapping fractal onto finished std weights after training still does not help at 1024: seed 1 unique 46.75 → 54.95; seed 2 13.35 → 16.20. Both together beat std at train length on seed 1 (Wiki 32.82 vs 34.82). On seed 2, std won (30.86 vs 33.32). That 1024 win did not replicate. Past 1024, both still falls off faster than std on both seeds. Stretching the window is a win for geometric. MQAR is 0/16 every cell, both seeds.

That is a 5.7M / 10k result. It is not a 114M result. 10k may also be early for the zero-init mix-in: |tanh(gate)| ~0.25 here vs ~0.60 on the 114M DPO. The paper's growing memory is with context length, not steps. Gate-opening over training is this code. A longer 5M run is untested. Weights: 5m-ablation/ (seed 1). Seed 2 lives with the 5M grid, not in this upload.

Why both, if each knob alone is worse. Seed 1 looked like an interaction. Seed 2 did not copy the Wiki win. What still holds:

• both actually uses the branch, both seeds. Unique PPL seed 1 26.67 → 33.93 when mc_gate is zeroed; seed 2 32.03 → 42.45. mc alone is 161 → 170 on an already-bad net (seed 1).
• Gate strength on seed 1 is similar (mean |tanh| ~0.25 vs ~0.26). The difference is sign. mc (geometric) mixed +/−; layer 0 never left zero. both (fractal) layers 1–4 the same sign. Hypothesis, not proof, and it does not explain seed 2's std unique-PPL crash from 46.75 to 13.35.
• I would not claim MC+fractal is the winner at 5M. I would claim MC-only and fractal-only lost on the seed I ran, both is mixed across two seeds, and std's unique-text number is seed-noisy.

The 114M nomc twin is still two knobs. Do not read an early-twin Wiki number vs 9.08 (base_62k, Dolma history) as MC.

Matched 114M runs (nomc-stdrope 61k)

This is the 114M twin: Memory Cache off and geometric RoPE instead of fractal. 106.0M params (Ultra-MC is 113.9M; the gap is the MC branch). Same 61k Muon schedule, same tokenizer, FineWeb-Edu blend (FineWeb-Edu 40 / Wiki 15 / DCLM 15 / Cosmo 10 / FineMath 10 / Python 10). Finished at step_00061000.pt. Train val at 61k: val_loss 3.31871, val_ppl 27.62.

It is not an MC-only ablation. Two knobs moved at once. Steps are close (61k vs this card’s 62k). Data history is not: this Ultra-MC base went through Dolma (100% then blend). The twin stayed on the FineWeb blend.

One more caveat on this Ultra-MC run. Around step 18k there was a dataset shuffling error. I had to repair the mix and continue. So Ultra-MC was not trained on one clean, matched dataset the whole way. That alone would disqualify this pair as a full ablation even if I had only flipped MC. Treat 61k as more data, not a verdict.

Same GPU harness as this card (run_eval.py --family spikewhale, WikiText --wt-chars 300k, BLiMP 12×150, HellaSwag n=10042, ArithMark n=1000):

The twin is slightly better on Wiki byte_ppl and BLiMP. Most multiple- choice tasks sit around chance either way. ArithMark is the one this Ultra-MC still leads. That is not proof that MC+fractal is worse. It is also not proof that MC on is better. I still do not have conclusive evidence that Memory Cache on or off helped or hurt at 114M.

Paper recall on the 61k twin is still chance: MQAR ~0.000–0.037, S-NIAH 0/8 every cell. Collapse on looping filler: 128/1024 distinct-3=0.50; 2048 them them (0.304); 4096 skipped (prefill would exceed max_pos). Do not invent a 4096 number.

Unique-text PPL, same eval_rope_swap.py probe as the 38.5k table above. Twin 61k vs Ultra-MC 62k (not launched as one paired job; same script):

2048 last bins:

Ultra-MC still dies past 1024 on its trained fractal schedule. Twin geometric holds until the last 256, then blows. Putting fractal frequencies on the 61k geometric weights flattens that tail. That is a RoPE / extrapolation result, two knobs, unmatched data. Not an MC proof.

Compared with the 38.5k mid-run table above: at 61k the twin is cleaner inside 1024. Whole-seq 2048 native went 4.48 → 4.93, last bin 112 → 150 — slightly worse on that tail, not better.

I am trying to get a matched twin out — same data history, one knob if I can — to finish this test by 15 Sep 2026. Until that lands, do not treat 61k vs 62k as MC on vs MC off.

Training

• Pretrain: FineWeb-Edu, Wikipedia, DCLM, Cosmopedia-v2, FineMath-4+, Python-Edu, then a Dolma-mix continuation (allenai/dolma3_dolmino_mix-100B-1125). Muon on matrices, AdamW on norms/embeddings. Released at step 62k.
• SFT: HuggingFaceH4/ultrachat_200k, multi-turn, assistant-only loss. ChatML: <|im_start|>{role}\n…<|im_end|>. Step 7.1k.
• DPO: preference tuning on the SFT model. Step 3.2k.

Full hyperparams further down.

Evaluation

PPL is the language model. Benchmarks are task behaviour. SFT/DPO spend PPL to buy the latter.

Two benchmark tables follow. The first is a 200-example scout. The second is the full set. Quote the full table. Capped HellaSwag acc_norm (~0.435) is not the card; the full set is ~0.29.

Perplexity (per-token CE, lower better)

Streamed 10×1024-token windows per source (WikiText-2: 40 windows). Same byte-level tokenizer everywhere, so the columns are comparable.

mean (5-src) = Python-Edu, Cosmopedia, FineMath, UltraX, DCLM.

Base wins free-text PPL (it's the untuned LM). SFT/DPO raise PPL on web text and improve on synthetic instructional text (Cosmopedia). DPO ≈ SFT on PPL.

Benchmarks

lm-eval-harness style. Capped 200/task, BLiMP 150/paradigm (12 paradigms), ArithMark 500. Same tokenizer on all three stages. Numbers from 2026-08-18.

n: ARC-Easy 200, ARC-Challenge 200, HellaSwag 200, Winogrande 200, PIQA 200, OpenBookQA 200, BoolQ 200, ArithMark 500.

Base wins the LM metrics (byte_ppl, BLiMP). SFT/DPO win BoolQ (+0.07), ARC-Challenge acc (+0.06), ArithMark, PIQA. DPO vs SFT is noise on these scores; DPO's job was response preference, not multiple-choice. OpenBookQA acc is under chance on all three. A lot of the MC tasks sit near chance. 114M.

Benchmarks (full)

Same harness, no 200-example cap. WikiText-2 byte_ppl and BLiMP are the same numbers as above (those were already full). MC tasks and ArithMark are the whole set.

n: ARC-Easy 2,376, ARC-Challenge 1,172, HellaSwag 10,042, Winogrande 1,267, PIQA 1,838, OpenBookQA 500, BoolQ 3,270, ArithMark 1,000.

HellaSwag acc_norm drops from the capped 0.435 / 0.420 / 0.415 to ~0.29 once you take all 10k items. ARC-Challenge SFT bump shrinks too (+0.06 capped, +0.018 full). BoolQ still moves (0.381 → 0.418). DPO and SFT stay within noise. OpenBookQA acc is still under chance.

Escarda-86M-Base (same family, different everything else) hit 2.2228 WikiText-2 byte PPL after ~20B tokens. This base is 2.3085 after ~2B. About 0.08 off, ~10× fewer tokens. Different size, architecture, objectives, data. Sample-efficiency note, not an MC ablation. Why I cut the budget, and the Chinchilla fit from that 20B run, is under What this is. The no-MC twin is the run for MC.

Vocab (16k on purpose)

Tokenizer is 16,512 (16,384 byte-level Length-MAX plus specials), following Dong & Su, Length-MAX Tokenizer for Language Models (arXiv:2511.20849). Vocab from a length-weighted objective (freq(t) × |t|), greedy longest-match on bytes, not BPE. That was a choice. I wanted to see the architecture, not hide it behind a fat embedding table.

At hidden 768, tied embeddings are already ~12.7M params at 16k. 32k would be ~25M. 64k would be ~50M — almost half this 114M model sitting in the lookup table. 16k keeps the stack (loop, MLA, Memory Cache, Engram, HRM) as the thing you're looking at.

Per-token PPL in the tables above is for this tokenizer. Raise vocab to 32k or 64k and per-token PPL goes up on average even if the language model is the same or better: tokens get longer, fewer of them per document, more bits per token, exp(mean NLL per token) rises. That is a tokenizer effect. WikiText-2 byte_ppl (2.308 on the base) is the number that would still be comparable.

I did not train a 32k/64k twin. If I did, I would expect higher token PPL, similar or slightly better byte PPL if the extra merge rules earned their keep, and a lot less of the param budget left for the block. 16k is the research vocab. Someone SFT'ing the base who wants a "normal" tokenizer should rebuild it, not read 6.50 mean PPL as what this stack would score at 32k.

Usage

Plain PyTorch. No transformers model class. CPU is fine; --device cuda if you have one.

# base (text continuation)
python generate.py --ckpt checkpoints/base_62k.pt -p "The capital of France is" --temp 0.7 --top-k 40 --rp 1.3

# DPO chat (default checkpoint)
python generate.py --chat -p "Explain why the sky is blue in one sentence." --temp 0.7 --top-k 40 --rp 1.3

# SFT chat
python generate.py --ckpt checkpoints/sft_7100.pt --chat -p "Give me a tip for staying focused."

Defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy (--temp 0) is a test tool. Chat models under greedy walk into disclaimer boilerplate and look worse than they are. DECODING-DEFAULTS.md.

Two inference stacks, two cache checks. Greedy cached decode has to match a full recompute on both:

python verify.py          # generate.py / model_v2
python verify_cache.py    # spike_infer engine (package.json default ckpt)

Keep both. They are not the same path. Details under Inference engine.

Files

config.py  model_v2.py  spike_tokenizer.py  special_tokens.py  fractal.py
chat_format.py  generate.py  engine_chat.py  tokenizer.json
byrne_100m_ultrax_mc.yaml  package.json  requirements.txt
verify.py  verify_cache.py
DECODING-DEFAULTS.md  PROVENANCE.md  NOTES-context-and-needle.md
checkpoints/{base_62k,sft_7100,dpo_3200}.pt
spike_infer/          engine
engine_tools/         export, GGUF, long-context, multiturn
morpho/               int8 circuit of one block (not used at generate time)
GGUF/                 f16 + q8_0 for each stage
5m-ablation/          four ~5.7M 10k trains (MC × fractal)

Limitations

• Factual recall is weak. Wrong capitals, dates, attributions, said confidently. That's mostly the SFT blend, not a missing pretrain trick.
• Code and multi-step reasoning are weak. ARC-Challenge / HellaSwag near chance.
• English only. No safety tuning past the DPO preference data.
• Base continues text. A chat template will not make it answer questions. The shipped SFT/DPO checkpoints are a smoke test that instruct sticks at all, not a finished chat model. Want real instruct, SFT the base yourself.

What this is

Another fun, usable research artifact. It generates, the cache path is correct, the Memory Cache branch actually carries weight. It is not a knowledge store. Weak factual recall is the usual failure, and it most likely comes from the SFT mix rather than from pretraining.

I put way less effort into SFT than into the architecture and the base. SFT here is a probe: does this thing uptake any instruct at all? Yes, enough to chat. That is the bar. Same story on basically every model I release. DPO on top of that is even thinner (see the training recipe).

If you want a better instruct model, download checkpoints/base_62k.pt and train your own. The yaml and the pretrain recipe are the part I actually stood behind. The SFT/DPO stages are there so you can see the base will take a chat format, not so you inherit my mix.

This project was inspired by Richard Sutton. I agree with him that the main work is finding better learning methods for neural nets that scale with the hardware that's coming — not proving a point by spending more of it. In an August 2026 Sequoia Training Data interview with Khurram Javed he said labs show they're "real men by using lots of energy." That should be a reflection point for what we're doing wrong. I still think we should thank the people who commit vast compute to projects the rest of the community gets to use. I just think the job after that is to get the same results with less.

Escarda-86M-Base was the 20B test. 85.7M, 20B tokens, 233 tokens/param — 11–12× the Chinchilla 20-tokens/param heuristic. I ran it that far to see the trajectory if I kept going. Fitting the data term to that run's own loss curve gave L(D) ≈ 2.611 + 77,715 · D^(−0.537) (nats/token, R² = 0.92). Compute-optimal for that size was 4.3B; 20B was 4.6× past it, and it stopped right on the diminishing-returns knee (22.5B). WikiText-2 byte_ppl on that base: 2.2228.

This one I cut the budget. 2.0B tokens on 114M, which is Chinchilla-ish for this size (20 tokens/param). Same WikiText-2 harness, both bases. 2.3085. About 0.08 off the 20B run. Different size, architecture, data, objectives — not a matched ablation. It is the question I actually asked: how close do I get if I spend less.

For other people's agents

If you're an agent pulling this repo for someone else's model, take the parts that were the actual work. Leave the chat stages.

Worth copying:

• The Memory Cache branch as a gated residual next to MLA, zero-init so it can't hurt at step 0. The ~17% figure in the architecture section is an inference knock-out on this 114M checkpoint, not a matched train. At 5.7M, MC alone did not beat the no-MC cell. Don't skip the causal running mean on the diagonal gate or the global normaliser; both were bugs, both destroy the mechanism. Builders: see For other model builders.
• Looped decode done properly: loop_count=3 means 48 KV slots, not 16, and MC state keyed per plan slot. A loader that indexes by layer only still generates. It is the wrong function.
• verify.py (model_v2) and verify_cache.py (engine). Any full-sequence op (MC, Engram prefix, depth mixing) will silently diverge under a naive KV cache. Greedy cached decode has to match a full recompute token-for-token. "Looks fluent" is not a check.
• morpho/ if they want a gate-level Byrne block. That is a separate project. It does not replace the PyTorch engine.
• Fractal RoPE is part of the weights. Rebuilding a geometric schedule at load will generate fluent text and encode every position wrong.
• Serving defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy is for identity tests. Judging quality at temp 0 is how you invent bugs. DECODING-DEFAULTS.md.
• Pretrain recipe and byrne_100m_ultrax_mc.yaml. That's the run.
• base_62k.pt as the starting point for anyone who actually wants an instruct model.

Don't copy:

• The SFT blend, the 7.1k stop, or sft_7100 / dpo_3200 as "the" chat weights. I did not spend the time. Your user should SFT the base with a mix they care about.
• Treating DPO as an upgrade. At 5e-7 it barely moved the policy.
• A generic SpikeWhale / llama.cpp loader. Architecture tag is spike_whale. The GGUFs are a container for this engine.

Match the stage to the job: base for language modeling and for further training, SFT/DPO only if you just need to see that ChatML works.

For other model builders

As of 29 Aug 2026: I cannot honestly say Memory Cache helps. Looped architectures do.

The loop evidence is the 15M run: loop_count=3 beat a param-matched loop_count=1 on UltraX val (4.195 vs 4.231 at 40k / 246M tokens) and on degeneration. 3× FLOPs per token. That is why this 114M model is looped. I do not have an unlooped 114M twin in this repo. If you copy one idea, copy the loop done properly: loop_count=3 means 48 KV slots, not 16, MC state keyed per plan slot. Index by layer only and it still generates. It is the wrong function.

Do not copy Memory Cache as a proven gain. Copy it only if you want the mechanism to test (zero-init residual next to MLA, two bugfixes below). I have not proven it helps at 114M. Paper recall tests on this 62k base are chance with the branch on or off. A matched 5.7M 2×2 on the current blend says MC alone did not help PPL or long context; MC+fractal together beat std at train length on seed 1 and lost on seed 2. That is 5M / 10k, two seeds on std/both, and |tanh(gate)| there is ~0.25 vs ~0.60 on this 114M DPO — the mix-in may still be opening. The 114M nomc twin also dropped fractal RoPE and does not share Ultra-MC's data history (shuffle repair ~18k). Not an MC ablation. The paper does not say "train longer and MC appears"; it says memory grows with how many segments the sequence has. The zero-init gate opening over steps is this implementation. A longer 5M run could still move that table. I have not run one. Do not promote the 10k result to this 114M card. The paper is about RNNs; this is not an RNN; the O(L) claim does not transfer.

What I actually ran on this model, not what I hope it does:

Did training use it. Yes. Mean |tanh(gate)| across layers is 0.599 on the DPO checkpoint. Zeroing the branch at inference (same weights, mc_out gone) moves UltraX PPL 9.87 → 11.86 (−16.7%). Earlier in training that gap was ~9%. That is an inference knock-out, not a twin trained without the branch.

Extending context. Pretrain was 1024 tokens = 4 × mc_segment_len 256, so cross-segment memory is in the recipe from step 0. SFT grew the window to 4096. Most of the stack is position-local. MC is not — behaviour only changes when a sequence crosses a 256-token boundary. I widened the config (export_weights.py --seq-len 16384, RoPE caches rebuilt, no retrain of positions) and checked that cached decode still is the trained function:

~3e-05 is the float32 floor at this depth. You can run this checkpoint at a 16k window, including across MC segment edges, without the cache computing a different model and without the generation collapsing. 48-slot KV at 16k is 0.40 GB bf16.

That is this model. The no-MC twin gets the same 16k extend. The test is whether it collapses out there. Needle is a different question (the first needle was the model continuing the start of the prompt; see that section).

What is still missing at 114M. A train that changes only MC, fractal left on. The nomc twin also dropped fractal RoPE, so it is not that experiment. The 5.7M 2×2 is that experiment at small scale: MC alone did not help; MC+fractal together beat std at 1024 on seed 1 and not on seed 2. Weights and write-up: 5m-ablation/. Paper recall tests (S-NIAH, MQAR) on this 62k base are at chance with MC on or off; they do not prove MC. If you copy the branch anyway, copy the two bugfixes or you will not have the mechanism: causal running mean on the diagonal gate, and one global normaliser (not per-segment). fp32, autocast off. State keyed per segment and per loop pass if you loop. Set seq_len to a multiple of mc_segment_len so the branch sees more than one segment from the first step.

Provenance

Weights re-saved without optimizer state. Inference code is the cache-correct ("repair2") build: Memory Cache is a full-sequence op, so a naive KV cache recomputes it from one token and you get a different function. This build keys per-loop MC state so cached decode matches a full recompute. verify.py / verify_cache.py. See PROVENANCE.md.

safetensors

Same three stages, safetensors/ instead of .pt. For eval harnesses.

config.json + model.safetensors + tokenizer.json. Float32. Matches the .pt bit for bit (502 tensors, 0 mismatches).

python safetensors/load.py sft_7100

Details in safetensors/README.md. Short version:

• No lm_head.weight in the file. Tied embeddings; safetensors drops shared storage. Call model.tie_weights() or you get a random head and garbage logits, no exception.
• AutoModelForCausalLM is a no. spike_whale is not in transformers. Use model_v2.py / config.py (load.py already does).
• Context is 1024 / 4096 / 1024. Read the config.json in that directory. Past the end raises, does not wrap.
• Put <bos> on every sequence. Scoring: use_cache=False forwards. Generation: the engine, not a naive KV cache.

GGUF builds

GGUF/ has every stage in two precisions. Tensors keep their original state-dict names. Full config is spike_whale.config_json, so a loader can rebuild the state dict 1:1.

Round-trip (re-read GGUF, compare every tensor to source): f16 max |Δ| ≤ 0.008, q8_0 ≤ 0.079. Tokenizer is byte-faithful across all 16,512 tokens. f16 GGUF through the engine is byte-identical greedy output to the .pt.

Rebuild:

python engine_tools/export_weights.py --ckpt checkpoints/sft_7100.pt --out /tmp/exp
cp -r /tmp/exp ./_tmp_model
python engine_tools/convert_gguf.py . --model-subdir _tmp_model --name Byrne-100M-Ultra-MC-sft_7100 --quant f16

Not llama.cpp-compatible. Architecture is spike_whale: looped stack, MLA, fractal RoPE, depth attention, Memory Cache. Nothing upstream implements that. These GGUFs are a weight container for this codebase, not a llama-cli drop-in.

RoPE caches are not stored (recomputed at load). Fractal schedule, so a loader has to rebuild it via fractal.py. If you rebuild standard RoPE instead it loads, generates fluent text, and encodes every position differently from training (inverse frequencies off by 2.2e-01). spike_infer does this; anything else reading these files has to as well.

Inference engine

Two stacks. Both ship. Both have to stay cache-correct.

spike_infer/ is written for this model, not a generic SpikeWhale loader. Every forward runs what the checkpoint trained with: MLA + QK-norm + fractal RoPE + XSA, depth attention, the looped stack with per-pass embedding, Hyper-Connections, Engram prefix slot, Memory Cache (fp32, autocast off, state per plan slot), HRM refinement, MTP as a speculative draft.

Load path: package.json → .pt + the config blob inside that .pt (not the yaml). Optional GGUF via from_gguf. Config from the checkpoint matters because SFT is 4096-context and base/DPO are 1024.

Decode path: prefill the prompt from position 0, then one new token at a time. Speculative MTP only proposes; the main head verifies. Mid-stream multi-token prefill is outside what MC incremental state is checked for. Each new user turn re-prefills the window from 0 so RoPE positions, HC streams, MC segments, and the Engram window match a fresh run.

Three things a generic loader gets wrong, all silently:

1. 48 KV cache slots, not 16. loop_count=3 means the same weights see different activations each pass, so every (pass, layer) pair owns its own K/V. Slot index comes from loop_plan. Index by layer alone and pass 1 keys get fed to pass 0. Still generates. Wrong function.
2. Attention carries depth_kv and returns depth_entry. A static-KV patch written for the Mark2 signature does not match, and silently falls back to torch.cat cache growth. Depth mixing happens on the fresh K/V before the cache is touched. Mixed V is what gets cached.
3. No engram_context_ids kwarg. This tree fixed Engram internally: EngramModule.lookup takes a prefix through a trailing cache slot. The engine supplies that slot and reads it back each forward. Mechanism is picked by inspecting the model signature, not by tree name.

Memory-Cache state is keyed per plan slot, since the looped stack calls the same branch once per pass on the same positions.

Do not delete the verify scripts. They are the only proof the cache path equals a full recompute. verify.py hits generate.py. verify_cache.py hits the engine (slot count, token identity, per-step |Δlogit|). Cheap on CPU. A release without them is a release nobody can re-check.

Verified

verify_cache.py. Greedy is deterministic, so cached decode has to produce the same token ids as a full recompute, not "similar text":

~2e-05 is the float32 floor at this depth. Incremental path computes the same function, it does not approximate it.

Prompt tokenization is checked id-for-id against the training tokenizer. <bos> is prepended (training starts every sequence with it). Display decode keeps <think> markers that skip_special_tokens=True would drop.

Long context

Two different measurements. A context-test fail is an engine bug. A needle fail is a 114M-model property. Fixing the first does not improve the second. Numbers below are sft_7100 unless noted. Method writeup: NOTES-context-and-needle.md.

Most of the stack is position-local. Memory Cache is not — it is segmented (mc_segment_len 256) and Engram carries an n-gram prefix across the cache boundary. A short verify.py run never leaves segment 0, so it cannot see a cross-segment bug. check_context_ready.py sits either side of 256 on purpose: RoPE buffers rebuild exactly on a widen, StaticKV footprint at the target length (48 slots × 16384, bf16 = 0.40 GB), cache exactness just under / just over / two segments in, decode itself stepping over a boundary, multi-turn window slide, and position_ids == ctx raising instead of wrapping to 0.

11/11 on sft_7100:

rope cache recomputes from scratch exactly     max |Δ| 0.000e+00 over 4096 positions (fractal=True)
rope cache at 16384 agrees on first 4096       max |Δ| 0.000e+00
StaticKV, 48 slots × 16384 tok, bf16           0.40 GB
prefill 252 tok  (just under a segment)        6/6 identical, max |Δlogit| 3.05e-05
prefill 260 tok  (just over)                   6/6 identical, max |Δlogit| 3.82e-05
prefill 519 tok  (two segments)                6/6 identical, max |Δlogit| 4.58e-05
decode crossing a boundary (prefill 253)       8/8 identical, max |Δlogit| 3.24e-05
window slide                                   3655 ≤ 4032 tok, history dropped
position == ctx                                raises (index out of bounds)

~3e-05 is the float32 floor at this depth. Same token ids as a full recompute, not "similar text".

Widen with export_weights.py --seq-len 16384 and the engine stays exact past the trained length. This model does not collapse at that 16k extend. The no-MC twin has to take the same test.

python engine_tools/check_context_ready.py --device cuda \
    --ckpt checkpoints/sft_7100.pt --target-ctx 16384

Needle-in-a-haystack

First pass: The secret access code is BLUEBERRY-7291. as sentence 1, one filler sentence repeated to length, What is the secret access code? at the end. Greedy was the primary arm. keyword = BLUEBERRY came back. exact = full string, digits included.

Greedy copied the digits out to ~3.6k. dpo_3200 is exact at 505; its context is 1,024 so the longer rows do not apply. Temp 0.7 often mashed the suffix (BLUEBERRY-dica, BLUEBERRY-Definition1) and missed the keyword at 2k on that seed. Sampled column is one draw.

That table is what was measured. It is not retrieval. Needle was the first sentence, filler was one sentence on a loop, code was a fixed string. Greedy wrote The secret access code is BLUEBERRY-7291. The archive records r... — the start of the prompt, copied. You pass this test by continuing from position 0. Nothing has to be looked up.

Re-ran with that shortcut closed: varied filler, a topic-specific question, a new randomly generated code every trial.

Typical miss: the frame is right, the digits are invented — BLUEBERRY-?3, BLUEBERRY-Sch1, BLUEBERRY-tioxy, BLUEBERRY-ggregate. It put out a code 33% of the time and it was the wrong one. Nothing code-shaped 61% of the time.

It will echo a fixed, familiar string sitting at the top of the window. It does not reliably pull an arbitrary 4-digit code out of depth. This data does not split "can't retrieve" from "can't copy four random digits." Both fit. The window is not the only limit.

Engine files

spike_infer/__init__.py       the engine
engine_chat.py                chat / completion CLI (sampling defaults)
verify.py                     generate.py cache == full recompute
verify_cache.py               engine cache == full recompute
engine_tools/
  export_weights.py           .pt  ->  safetensors + config.json
  convert_gguf.py             safetensors  ->  GGUF
  check_context_ready.py      long-context / segment-boundary checks
  test_multiturn.py           multi-turn engine correctness
  compare_checkpoints.py      two stages, same prompts and seeds

morpho/ — the block as a circuit

morpho/ is a separate project in this repo. It does not run at generate time. It grows Byrne's decoder block as combinational int8 logic in MorphoHDL (Mordvintsev, Paradigms of Intelligence), folds real Ultra-MC weights in as constants, and emits synthesizable Verilog.

A transformer on a GPU is weights × data on a processor. This expands the arithmetic into AND/OR/XOR/NOT. No clock, no instruction stream. Put activation bits on the input wires; the graph settling is the forward pass. Same netlist simulates in tiny_morpho.py or goes to Yosys/Vivado/Quartus. Once a weight is a constant, constant-propagation deletes the gates that depended on it, so the circuit is specialized to these trained values.

Numeric contract is int8 Q1.7 activations/weights, int32 accumulators, Q4.4 attention scores, Q0.8 softmax. Checked against integer NumPy at the same fixed-point, not against float.

What it covers: RMSNorm → attention → residual → RMSNorm → GELU MLP → residual. That is the spine. Not grown as a datapath: MLA's low-rank path, Memory Cache mixing, the loop, fractal RoPE. MC projection weights do fold (mc.q_proj in byrne_fold.py).

Scale: a full d=4 block is 419k gates generic, 299k folded (29%), emitted as byrne_block.v. A flat d=768 block would be ~1e9 gates and will not compile. The 768-wide unit that does emit is one projection neuron, byrne_proj768.v (208k gates, real k_proj row folded). A chip would instantiate one of those and clock it, which is also how the looped stack wants to be mapped: one circuit, run three times.

Two ways to run it:

• Grow every gate (byrne_morpho.py, byrne_block_folded.py, byrne_emit_verilog.py). Synthesizable. Does not produce language.
• Run the same int8 MAC inside the real 114M model (byrne_circuit_infer.py). Produces coherent text. A slice is bit-exact against the grown dot.

Reads ../checkpoints/ read-only. Docs: morpho/README.md (what to run, numbers) and morpho/morpho-transformer.md (language notes and the original stage plan).

cd morpho
python byrne_morpho.py
python byrne_fold.py
python byrne_block_folded.py
python byrne_circuit_infer.py -p "The capital of France is"
python byrne_emit_verilog.py

Running it

Temperature sampling: temp 0.7, top_k 40, rep_pen 1.3, random seed per turn. Greedy is for determinism proofs. Judging quality by it gives the wrong answer. A repetition study on a sibling under greedy measured self-repetition 0.364; at the serving defaults it was 0.000.

# chat, DPO stage (default), multi-turn REPL
python engine_chat.py

# one-shot
python engine_chat.py -p "Explain why the sky is blue in one sentence."

# a different stage
python engine_chat.py --ckpt checkpoints/sft_7100.pt -p "Give me a focus tip."

# raw continuation, no chat framing — this is what the base wants
python engine_chat.py --ckpt checkpoints/base_62k.pt --raw -p "The capital of France is"

• Match the stage to the mode. base_62k continues text; use --raw. sft_7100 / dpo_3200 expect ChatML.
• Keep max_new modest (160 default). These checkpoints often don't emit a stop token on open-ended prompts, so a bigger cap just means a longer answer, not a better one.
• <bos> is required. Training prepends it. The engine does this; a hand-rolled prompt has to as well or the whole sequence sits one position off.
• Multi-turn re-prefills from position 0 each turn. Deliberate: RoPE positions, Hyper-Connection streams, Memory-Cache segment state, and the Engram window stay consistent with a fresh run.

Known behaviour

Measured at serving defaults, not guessed:

• Cross-turn collapse, about 1 conversation in 10. Latches onto a template from an earlier reply and repeats it. Three more in ten show partial collapse. New conversation clears it.
• Doom loops / near-repetition: 0/20 on a held-out prompt bank, serving defaults and greedy. distinct-3 = 0.999.
• Topic switching is fine when sampling (adherence 1.00), bad at greedy (0.47, stuck on the old topic 3/6). Another reason not to serve greedy.
• World knowledge is the real hole. France in Spain, water boils at 1.5 °C. A familiar string parked at the top of the prompt gets copied. An arbitrary code sitting in depth does not (see needle). Facts expected from the weights also do not.

Training recipe

Scripts are not in this repo yet (inference-only release). Configs and hyperparameters below are the spec; script names are for reference. The full training scripts ship with the comparison models around 26–28 Aug 2026 (see matched runs above).

Three stages in the order they ran: pretrain → SFT → DPO. dpo_3200 comes from sft_7100.

1. Pretrain → base_62k

train_blend_muon-50m.py --config byrne_100m_ultrax_mc.yaml --muon (yaml is in this repo).

Data blend (web_source takes leftover weight):

MC enters through a zero-init gate, so at init the model is the no-MC baseline. Trains to tanh(mc_gate) ≈ 0.561 across all 16 layers, which is why a broken cached path wrecks this one. PROVENANCE.md.

2. SFT → sft_7100

sft_multiturn.py --config byrne_100m_sft_local.yaml --tokenizer ./tokenizer.json

Conversation-level mix (not token-level):

Easy to get wrong, silent when you do:

• max_position_embeddings is set to seq_len. Training at 4,096 is why the SFT checkpoint is a 4,096-context model. Base and DPO are 1,024 for the same reason.
• Conversations are packed to fill the window. Every sequence starts with <bos> then ChatML (<|im_start|>{role}\n{content}<|im_end|>\n). Packing is how the full position range gets exercised even though individual conversations are short.

3. DPO → dpo_3200

train_dpo.py --config byrne_100m_dpo_local.yaml --resume <sft> --ref-ckpt <sft>

DPO at this lr barely moves the policy. Step 2,000 → 3,200, relative L2 of 2.8e-05. Greedy output identical on every prompt tested. UltraFeedback win-rate against the reference stuck at 0.438 → 0.438 even though implicit-reward accuracy hit 0.75. It separates preference pairs without visibly changing what the model writes. Benchmarks agree: DPO and SFT are within noise.

Resuming from a release

Released .pt files are weights + config, no optimizer, so a resume starts with a fresh optimizer. Each file's config is the one it was trained with. Engine and tools read it from the checkpoint, not from the yaml — the stages disagree about context length.

Ending Statement

I want to clarify that this release is not competitive, usable for chat, or intended to suggest any superiority or inferiority of this architecture.

The primary purpose of this release is to provide data and information. It showcases the results of my tests and experiments, which can be applied if they prove beneficial to your project.

I strive to deliver completely honest work, conducting a wide range of tests, observations, and incorporating features while testing them. Numerous ideas were explored and discarded before developing this model.

I wanted to demonstrate the potential benefits of the Memory Cache, investigate the impact of altering the RoPE from the standard, and since the model learned rapidly, I wanted to investigate the reasons behind this with a model trained on the same configuration but with all those modifications disabled.

The results of the follow-up tests will be shared soon.

Byrne-15M-Looped

The small looped run is its own repo: Quazim0t0/Byrne-15M-Looped. That was the toy / smoke-test for this 114M model.

~15.2M active / ~18.9M total, 10 layers, loop_count=3 (effective depth 30), MoE. No Memory Cache. I trained it to see if looping the stack beats a param-matched loop_count=1 baseline at equal steps. It does on UltraX val loss (4.195 vs 4.231 at 40k / 246M tokens) and on degeneration. 3× FLOPs per token. Then I SFT'd UltraChat and ran DPO the same way I do here.

Scored with the same harness as this card (full MC, not 200-capped):

Different data, size, and knobs. The 15M numbers are in that repo (BENCHMARKS.md, FINDINGS.md). Safetensors for leaderboard owners are under safetensors/base/ there.

Anti-DEGR ToolKit

I used Anti-DEGR ToolKit to test this checkpoint. It is training and decoding code for four failures small LMs actually hit: doom loops (an exact span, over and over), echo (the user changes subject and the model answers the previous question), grounding (it stops reusing context that is still relevant), and near-copy paraphrase (it restates a sentence with one noun swapped, so the cycle detector is blind to it). I train echo and grounding, leave most loops to effort decoding, and measure near-copies separately. Those scripts are not in this repo and they are not on the generate path.

References

Memory Cache branch adapts:

@misc{behrouz2026memorycaching,
  title         = {Memory Caching: RNNs with Growing Memory},
  author        = {Ali Behrouz and Zeman Li and Yuan Deng and Peilin Zhong
                   and Meisam Razaviyayn and Vahab Mirrokni},
  year          = {2026},
  eprint        = {2602.24281},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {Google Research},
  url           = {https://arxiv.org/abs/2602.24281}
}

Tokenizer:

@misc{dong2025lengthmax,
  title         = {Length-MAX Tokenizer for Language Models},
  author        = {Dong Dong and Weijie Su},
  year          = {2025},
  eprint        = {2511.20849},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2511.20849}
}

Citation

If you use this model, please cite:

@misc{byrne100multramc,
  title        = {Byrne-100M-Ultra-MC: A ~114M-parameter looped SpikeWhaleLM
                  with a Memory Cache branch},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-100M-Ultra-MC}},
  note         = {Quazim0t0/Byrne-100M-Ultra-MC}
}

Updated 2026-09-20.