Surgical Pruning of Qwen3.8-Flash-Next (512→128 Experts) and Full Post-Training Pipeline

One upstream MoE, six generations of masks, one bit-exact surgery, a rebuilt factual memory, and a three-stage post-training (SFT → SimPO → RLVR) — with every error I made and every idea I falsified along the way.

This repository is the complete technical record of the project: the activation-instrumentation artifacts, the mask-generation methodology, the surgery code, the PLE (n-gram memory) rebuild, the trainers, and the full lab notebook. The champion weights live in sibling repos (model map).

TL;DR

What I set out to do: take Qwen3.8-Flash-Next — an ~180B-class MoE (~130B of core network + 51B of n-gram memory table + 4B MTP) — and cut it by 75% of its experts (512 → 128) while conserving the maximum knowledge possible: protecting the model's grammar (tags, syntax, formats) perfectly, and paying for the size reduction with a deliberate sacrifice of the long tail — rare languages, niche domains, a few entire areas of expertise — instead of letting damage fall at random. Then post-train the result into a competent, honest reasoning model. Budget: about $300 of rented GPU time.

What I actually got: a 40B model (S6-K128-P4) with one genuinely excellent feature — the writing. The reasoning prose is structured, self-correcting, formally organized; it writes valid code, dispatches tools canonically, and argues back when you assert falsehoods. Everything else is weak, and I say so plainly: the benchmark scores are bad (GPQA ≈ chance, MATH-500 17.8%, IFEval prompt-strict 37%), factual recall took the hardest hit, and the blame splits — an extremely aggressive prune, my own heal/finetune errors (a factual memory that silently loaded random noise during the mid-lineage training stages, a SimPO dataset that structurally could not learn, a first mask that was statistically random), possibly the Thinking Cap method itself — the thinking format every trace passes through (this project's biggest achievement) is a named, un-ablated suspect (§8.3) — and measurement artifacts (format parsers, budget-exhaustion empties). The control experiments that would apportion the blame precisely — the unpruned upstream, the full dense-27B comparison — were never finished: the money ran out at ~$300. Every error is documented in §5; good and bad traces sit side by side in §8.2.

What this proves anyway: losing 75% of the expert mass does not produce a broken model. The damage is selective and mappable — language, logic, code structure and tool use survive; trivia and arithmetic robustness don't. So massive expert pruning is a viable technique with a known cost schedule. And there is clear headroom for a v2: the K=192 mask keeps +14 points of routing mass, the factual-table pipeline is now fixed and measured, and every failure mode has a named fix.

Why this architecture was the perfect guinea pig: only ~2% of the experts (10 of 512) fire per token per layer; the routers that make every decision are just 15.7M parameters (0.039% of the network); and about a third of the whole model was a lookup table — stored memory, not compute. In fact the 75% cut saves memory, not FLOPs: evaluating 10 experts out of 128 costs the same per token as 10 out of 512.

---

☕ Did the artifacts, the methods, or the honest failure list earn a coffee? → Buy me a Ko-fi — every ☕ = more pod hours, and the v2 (K=192 + clean heal) is already designed and waiting.

🚀 Entering the LLM fine-tuning / inference world yourself? Start with my RunPod referral → runpod.io?ref=ssakdva8 — you pay exactly the same, the project gets a boost, and §9 (Toolbox) gives you everything to replicate this pipeline step by step.

💸 For scale: the whole saga — 9 days, ~730 GB in total, 17 repos consolidated into this one — $300.

---

Model details

Intended use, out-of-scope uses, and limitations

This is a research model. It exists to be studied, not to be used. It is a laboratory artifact for investigating what survives a 75% expert prune, how a factual-memory table changes training dynamics, what breaks in a post-training cascade over a pruned MoE, and where exactly the damage lands (§8.3). Nobody should deploy it as a product, an assistant, or any system with real users — it is below the quality bar for that, by its own accounting (§8.3). What it is good for:

• Pruning research: reproducing or extending the mask generations (§2.3), the bit-exact surgery, the C1–C6 verification framework — or testing whether the K=192 headroom converts into quality.
• Post-training research: the SimPO iso-density finding (39.3% → 80.7% from dataset curation alone), the verified-reward loop, the RFT anchor cascade — all runnable from the adapters and datasets published here.
• Architecture study of qwen4_exp: one of the few places where this hybrid architecture (GDN/QSA + MoE + external n-gram memory) is instrumented, served and documented end to end.
• Reasoning-behavior observation: the five thinking tiers (off/low/mid/high/xhigh, selected via the system prompt) modulate deliberation depth measurably, and the model's strong suit — structured, self-correcting reasoning prose — makes its traces unusually readable for analysis.

Out of scope.

• Any production or end-user deployment — this is a research artifact, full stop.
• Encyclopedic recall without retrieval: GPQA Diamond ≈ chance — this model trades factual trivia for size; use it with RAG/tools for facts.
• Unassisted exact arithmetic: multi-digit multiplication without a scratchpad is a limit of the whole architecture class (TC⁰), aggravated by the prune; delegate to a code interpreter.
• Quantized serving: BF16 only (see §8.3).
• High-stakes deployment without verification: the model can produce verification-shaped text that is itself wrong (§8.3 #3); keep graders/tools in the loop.

Limitations. The honest failure list and its forensics live in §8.3; the benchmark scores above carry format-parser caveats documented in §8.2.

Get started

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Davd-b01/qwen3.8-flash-next-40b-prune-research"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content":
     "Reasoning effort is set to high. Prove claims rigorously, verify intermediate "
     "derivations, confirm arithmetic carries step by step, and test edge cases before "
     "finalizing the result. Prioritize quiet correctness over rhetorical emphasis."},
    {"role": "user", "content": "Prove that sqrt(2) is irrational."},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048, temperature=0.7, top_p=0.95,
                     repetition_penalty=1.05, do_sample=True)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Notes: (1) the champion loads natively — no patch script (the factual table is embedded in the fused parameter); the foundations (S6-K128-P0) require the cargar_tabla_ple patch (§11). (2) For serving, SGLang gives the best throughput on this architecture — recipe in §11.

---

Table of contents

• TL;DR
• The story, in plain words
• How everything is named (the convention)
• The nine-day campaign (process timeline)

1. The target: Qwen4Exp anatomy
2. Pruning methodology
   • 2.1 Calibration corpora
   • 2.2 Activation instrumentation — the signal artifacts
   • 2.3 Mask generations S1 → S6
   • 2.4 The surgery (bit-exact expert slicing)
   • 2.5 The router-truncation problem
3. The PLE n-gram memory
4. Post-training
   • 4.1 Data factory (Best-of-3, TCS tiers)
   • 4.2 SFT — asymmetric DoRA
   • 4.3 SimPO — the iso-density fix
   • 4.4 RLVR — verified rewards, offline
5. The error ledger
6. Rejected ideas, with falsifications
7. Academic foundations
8. The 75% thesis: competence survives
9. Toolbox: the reusable pipeline, script by script
10. Artifact map
11. Reproduction

• Training & compute accounting (environmental impact)
• Citation
• Support this work
• Contact
• Appendix: the math that runs this model

---

The story, in plain words

(No jargon. If you read only this section, you will understand what happened and why it matters.)

Start with the shape of the problem. Imagine a sheet of paper, handwritten edge to edge — every square centimeter used: code, math proofs, a dozen languages, legal doctrine, tool syntax, conversation, science. That is what this model is: ~180 billion parameters of densely-written knowledge, kept in 512 interchangeable "experts" per layer, of which only a team of 10 reads each word. The challenge I set myself: shrink that sheet to a quarter of its size — and keep it legible. Most of the content had to survive; the grammar — the tags, the syntax, the formats — had to survive perfectly; and some regions, knowingly, would be sacrificed to fit the budget: the rarely-visited margins, the long-tail languages, a few entire areas of expertise.

The pruning form of that bargain: erase 384 of every 512 experts, choosing them so every region of the sheet keeps enough of its text, and retrain just enough for the surviving lines to cover what was removed. And one tempting shortcut was off the table from the start. The literature says MoE models carry many duplicate experts — redundant copies you can merge away, free space. I measured it: barely 0.12% of expert pairs here are near-twins. This sheet was never written twice. Every line removed was the only copy. That single measurement turned the project from an organizational problem into a genuine sacrifice — and dictated the whole design: conserve the most-used knowledge perfectly, protect the grammar at all costs, and choose the sacrifices deliberately instead of letting them happen at random.

How do you decide which lines to erase? Not by their ink alone — and here the record owes a plain confession: my first attempts failed because I wanted to be cheap. Rather than rent GPU time to watch the real model work, I took the tokenizer's embeddings and half-passed them through the layers to see which experts "lit up" — a proxy of a proxy, tried to save a few dollars of teacher forwards. It was a complete error: the first shipped mask agreed with real routing on barely 30% of experts — worse than a coin flip, because it carried the false confidence of looking principled. What works is watching the sheet being read: run the original model over a deliberately balanced buffet of tasks (code, math, science, 17 languages, tool calls) and record, token by token, which experts fire and how much they matter. Then keep the 128 that carry the load — making sure no region (no task type) loses all of its text. That measurement, not the cutting, is the heart of this project, and the measurement ledgers are published here.

The cutting itself is done so that nothing is recomputed: the 128 survivors keep their exact original weights, bit for bit, and everything that is not an expert — the vision system, the draft head used for speculative decoding, the embeddings — passes through untouched.

Then the sheet needs physical therapy, for two reasons. First, the routing brain still expects 512 experts, so it must relearn — and it cannot relearn alone: experts that are never chosen receive zero learning signal (I tried routers-only training; it failed cleanly). So the routers train together with a light adapter layer across the network. Second, the model's "factual memory chip" — a 10 GB lookup table that injects facts — had to be rebuilt from 11% useful to 90% useful, because the original had been hashed with the wrong keys.

Along the way came the two best bugs of the project, both instructive:

• The memory chip was never plugged in. For days, every training run and every benchmark loaded a random-filled table, because the file stored the memory as 128 pieces while the loader expected one piece — and neither side complained. Different machines behaved differently because each one received a different random table. Found by checking a single number (row magnitudes) against the spec, for $0 of GPU time.
• The validation that validated itself. An earlier rebuild of the memory chip was certified "verified" by a test that used sentences from its own textbook: 100% on the test, 11% in production. Test data must be disjoint by construction, not by intention.

What survived: reasoning, code structure, tool use, formal logic, and the refusal to flatter the user — all intact, all improved by a three-stage finishing school (first teach format, then teach preference, then drill arithmetic with a strict automated grader). What got hurt: encyclopedic recall (the model kept its methods and lost much of its trivia) and mental multiplication (a limit of this entire model family, aggravated by the prune, fixed on the training bench by teaching a show-your-work habit). The final model is coherent, writes valid code, uses tools, and every one of its weak circuits is documented — because knowing exactly what was lost is what makes the 75% cut defensible.

---

How everything is named (the convention)

The historical names collided: "v6" meant a mask generation and a training run and a repository; "v4" meant both an old mask era and an old trained model. The convention below separates the axes so every artifact has exactly one code.

Rule: every artifact is <AXIS><generation>-<qualifiers>. Five axes:

Surgery generations (the mask lineage — §2.3):

Post-training generations (§4):

Factual table generations (§3):

Calibration corpora (§2.1): CAL1 = m100k (the 17-language traffic census), CAL2/CAL3 = GATE-DS v1/v2 (the balanced contribution census, with and without a reject-domain).

Full model codes — composed as S#-K<size>-P# — now mapped to paths inside this single repository (the 17 legacy repositories were absorbed here and deleted after hash-verification; lineage/MANIFEST.md records the ones deleted without copy):

One deliberate exception: published filenames keep their legacy labels (mascara_v4_m100k_K128_lam005.json, dora_adapters_rlvr_final.pt, …) so that every number in this README remains traceable to a literal file. The S/P/PLE/CAL codes are the lineage layer on top, not a rename of the artifacts.

---

The nine-day campaign (process timeline)

Everything above happened in nine days of pod time. The record, so the process is legible as a sequence of decisions rather than a monolith:

---

1. The target: Qwen4Exp anatomy

Qwen/Qwen3.8-Flash-Next is a hybrid MoE ("qwen4_exp" architecture, transformers ≥ 5.17) with a fused external factual memory. The pruned derivative I ship is a 40.22 B-parameter model. Of those, ~2.36 B of routed parameters fire per token (top-10 of 128 experts, 48 layers) on top of the always-on dense subsystems — 2.09 B hybrid attention, 1.27 B embeddings + lm_head, 641 M hyper-connections, 236 M shared expert, 16 M routers, ~33 M PLE projections — for ≈ 6.6 B active parameters per token. The 5.15 B PLE table is stored memory, not compute: 16 row-gathers of 160 dims per token, zero FLOPs:

Layer pattern: 3 GDN : 1 full self-attention (QSA, sparse indexer 4×/2048). Factual memory injection happens in layer 2 via deterministic n-gram hashing → table gather → temporal conv → SwiGLU-gated residual injection (see §3). A few numbers worth internalizing before the methodology:

• The upstream was already mostly idle capacity. 512 experts per layer, top-10 routing: ~2% of the experts evaluate each token.
• Management is microscopic. The 48 routers that decide everything weigh 15.73M parameters — 0.039% of the model. (Which is why "just retrain the routers" is both tempting and impossible — §6.3.)
• A third of the model was a phonebook. 51.2B of ~175B parameters (~29%) were the n-gram table — stored facts, zero FLOPs. In the 40B it is still 5.12B (12.8%).
• The 75% cut saves memory, not compute. Active parameters per token (≈6.6B) are essentially unchanged from upstream — the win is 180B → 40B of stored weights, which is what makes local serving and quantization conceivable at all.
• Upstream had ~96 nearly-dead experts per layer. Capturing 95% of routing mass needs K≈416 (the mass curve of §2.3); the tail beyond that rides almost no traffic.

The checkpoint also carries a vision tower (333 tensors, Qwen2.5-VL) and an MTP draft head (31 tensors, 512-expert MoE — deliberately NOT pruned, it is the speculative-decoding draft); both are pass-through, never instantiated by transformers (text-only serving), preserved bit-exact in every published repo.

---

2. Pruning methodology

when I started, I thought this would be easy. On paper it was very clear: pick the best 128 experts, prune them, do a heal, package, go home. Oh boy, it was not. Before touching a single weight I had to learn which parts of the original model actually do the work — by instrumenting it while it reads a balanced buffet of tasks — and only then decide what to remove. And the cutting itself turned out to be the boring part: survivors keep their exact original bits; everything hard lived in the measuring, the memory chip, and the finishing school.

2.1 Calibration corpora

Two corpora, built to answer different questions:

• m100k — 198 rows / ~111.5K real tokens (30 strata by set-cover of markers, ChatML, 17 languages, floor 2.5K / cap 8K tokens per row). Purpose: expert traffic profiling of the already-pruned v3 model (§43) and the first real-activation mask (v4).
• GATE-DS — a deliberately balanced corpus so that every expert gets invoked: v1 = 24 rows / 27,881 tokens / 12 strata; v2 = 46 rows / 53,877 tokens (keep-domain + reject-domain + 16 languages + uncensored split). Purpose: contribution measurement under guaranteed coverage ("Blind Collector Defect": with E=512, k=10, a dataset that activates <95% of experts assigns zero saliency to never-invoked experts and amputates uninvoked capabilities — Chernoff–Hoeffding gives the required ≥5.4M-token budget for ε=0.05; my strata quotas are the practical fix).

Domain-level PPL of the v3-dedup model on m100k (pre-heal): swe_coding 3.58, math 5.48, science 5.48, code 6.30 — vs degraded hf_aya_jpn 7,532 (the multilingual tail was deliberately sacrificed; see §3.2).

2.2 Activation instrumentation — the signal artifacts

All mask generations from S4 onward (legacy v4) are computed from recorded activations of the original 512-expert base model, not from weight statistics. There is only one base model (Qwen/Qwen3.8-Flash-Next); it was captured through its official FP8 release purely because the BF16 weights (~360 GB) do not fit on any single GPU, while FP8 fits on one H200: 125B + 51B n-gram + 4B MTP on a single H200 (141 GB) with: dtype="auto" (a torch_dtype=bfloat16 would upcast to 360 GB), PLE offloaded to CPU RAM (102 GB), FP8 kernel shim (kernels==0.16.0), max_memory 92% — 144.2 GB VRAM, 168 s load. The full operational recipe (including the 20 errors that taught it) is in docs/bitacora/RECETA-POD-H200.md, and the pod dissection playbook in docs/bitacora/GUIA_OPERATIVA_DISECCION_PODS_H200_RTX6000.md.

Signals recorded per forward (accumulator np.savez_compressed every 25 rows, crash-safe):

Final instrumentation artifacts (all committed to HF, see §9):

• ckpt_state_final.npz (98 MB) — hits + coact + the 34 gates captured by hooks (14 layers were disk-offloaded; hooking meta tensors is impossible, hence the safe_open pass).
• gate_w_full.npz (118 MB) — all 48 upstream routers (48×512×2560, bf16, compressed).
• senales_contribucion.npz (86 MB) + senales_v2.npz (88 MB) — REAP-style saliency S_e = (1/|X_e|) Σ g_e(x)·‖f_e(x)‖₂ over GATE-DS v1/v2.
• mascara_*.json — every mask generation (schema below), mmr_sweetspot_report.json, mmr_ranked_lists.npz, barrido_fino.json, n3_metrics.json, n3_hits.pt.

A full user guide for these artifacts — byte-verified schemas, capture method, the reproduction script, claim→artifact traceability, and community analysis ideas — is in docs/ACTIVATION-ARTIFACTS-GUIDE.md. The companion analysis (pipeline/08_analisis/analizar_activaciones.py, ~3 min of CPU) recomputes every selection number from the NPZs — the mass curve (51.8/66.1/77.6), the ×7.8 expert-usage lift, the 0.11% gate near-twins, the 1σ ordered-noise zone — and adds findings never measured before: routing entropy of the teacher is 8.58 of 9.00 bits (95% of maximum — over-dispersion measured directly); only 14 of 24,576 expert-slots were never selected (the waste is diffuse under-use, not dead experts); and 76.5% of expert pairs co-fire at least once (top-10 teams are real but loose).

Throughput engineering of the instrumentation itself: the m100k teacher pass streamed 108,580 tokens in 2,296 s; the GATE-DS pass added three optimizations — top-k computed on raw logits, the softmax evaluated only over the surviving top-10, and np.add.at vectorized scatter for the stratum accumulators — 24 rows (27,881 tokens) in 471 s = 52.5 tok/s, 3.5× over naive. Layers disk-offloaded under device_map cannot be hooked (meta tensors), so their W_gate came from a separate safe_open pass over the shards — lossless, because a router weight is static and (in the upstream FP8 export) never quantized. Note the FP8 release was used only as a measurement rig: the surgery and every shipped weight came from the orcarouter BF16 source — chosen by design, not convenience (§2.4).

Mask file schema (two generations, both forward-compatible):

// v3/v4 schema
{ "layer_masks": { "0": [int, ...], "1": [int, ...], ... } }          // 48 × K expert ids

// v6 schema (the one the surgery consumes)
{ "layers": { "layer_0": { "selected_experts": [int, ...] }, ... } }

2.3 Mask generations S1 → S6 (legacy v1 → v6)

Each generation is kept because each one falsified something specific.

The regime that shaped everything (understood in hindsight, TESIS-PODA-V4): with 512 experts of very fine granularity (intermediate/hidden ratio ≈ 0.23, finer than DeepSeekMoE's 0.69) and k/E = 10/512 ≈ 0.02, this model routes in the over-dispersed regime — the literature documents that standard importance signals (frequency, aggregated hits) collapse exactly here, because aggressive load-balancing flattens routing until neighbor ranks are statistically indistinguishable (§2.3's 1σ finding is that law measured on our own data; the direct measurement: routing entropy 8.58 of 9.00 bits, hottest expert holds just ×7.8 the uniform share, 14 of 24,576 expert-slots never selected — see the artifacts guide). The operator's principle behind the final criterion: "everything is a sacrifice — we select a few in exchange for what we don't serve."

S1 — split-repack (legacy v1; superseded). Split each 640-neuron expert into 2×320 sub-experts (exact additive decomposition of SwiGLU FFNs, DeepSeekMoE Thm. 1), prune to 256, repack 128×640 by cosine-affinity pairing. Valid math, but produced emergent auto-fusions [2k, 2k+1] and was abandoned for whole-expert selection in every later generation.

S2 — weight statistics (legacy v2). Router row-norms + embedding projections, CPU-only. Weight-space methods agree with each other — the embed-proxy pilot recalls 0.738 of the S2 mask — but both disagree with actual routing: against the forward-derived reference the recall is 0.228. Lesson: at 512→128 the value is in the ranking, not the retention — every reasonable method retains ~25% of mass; which 128 you pick is everything.

S3 — embed-space knapsack (legacy v3). Born of a false economy: my attempt to avoid paying for teacher forwards — pass the tokenizer's embeddings through the layers and watch what lights up. Download only the 48 routers (~315 MB) + embeddings; proxy signal E[token] @ W_gate.T → top-10 over CAL1; dual-Lagrangian knapsack with stratum quota. Retention 25.2% uniform across layers. Dedup variant (S3d, legacy v3-dedup, MMR greedy, score(e) = s(e) + 0.5·n(e) − g(e) − λ*·max cos(e,s) + Q·[deficit coverage], λ*=1.0) maximized intra-set diversity at −0.01pp mass cost. Gates V3 (orphan-specialist) and V5 (null-model) passed. This is the mask that produced the first pruned model — and it was, in effect, random: post-hoc measurement on real forward activations gave M_kept = 29.42% routing mass (random-128 baseline ≈ 25%), with 1,671,932 lost-specialist events (token×layer ×pruned-expert at >2.5σ), and only 30.26% overlap with what real routing prefers. Root cause (TESIS §2.2): with 52.1M routing slots, 94% of neighbor rank pairs in the decision zone (64–256) are statistically indistinguishable at 1σ — the mask's decision boundary is ordered noise. A structural note that reframes the whole dedup effort: the literature's usual free lunch — MoE models carrying many duplicate experts to merge away (the premise behind dedup-based pruning elsewhere) — does not exist in this teacher. Only 0.12% of the 512×512 gate pairs per layer exceed cosine 0.8 (0.026% above 0.9): the upstream is nearly orthogonal, so pruning here could not be "merging copies" — every removed expert was the sole carrier of whatever it knew. The dedup machinery (§6.1) was built for a redundancy this model doesn't have. Quantified later as the mass curve @128 = 51.8%, @192 = 66.1%, @256 = 77.6%, K95 = 416 (CAL1, λ=0): compressing to 1/4 the experts without retraining cannot capture more than ~half the routing mass — everything else must be healed or lost.

S4 — real activations of the 512-expert teacher (legacy v4-m100k). Forward of the full upstream over CAL1 (108,580 tokens, 2,296 s), signals as in §2.2, MMR with running-max ranked lists (any K is a prefix). Top-128-by-hits mask (mascara_v4_m100k_K128_lam005.json, sweet-spot sweep K∈{128,192,256} × λ=0.05).

S5/S6 — contribution signals on the balanced census (GATE-DS) (legacy v5/v6). Three selection criteria compared on identical signals (K=128):

Signal definitions, so the criteria are reproducible from the §2.2 accumulators. For expert e in layer l over a corpus partitioned into keep/reject rows: m_e (Σw̃) is the rank-weighted routing mass, m_e = Σ_rows Σ_{j≤10} w̃_j·1[e = top_j(row)] with w̃ the top-10 softmax — this discounts an expert that is merely rank-10 often in favor of one that is rank-1 often. keep_e/rej_e restrict the sum to keep-/reject-domain rows; v6disc ranks by keep_e − λ·rej_e (the only criterion that actively pushes reject-domain traffic out: λ=1.0 cuts reject-leak 63.0% → 43.6% at equal keep-mass, 69.8 → 69.5). Redundancy terms: gate-cosine cos(W_gate[:,e], W_gate[:,f]) — the only redundancy signal that behaved as advertised — and Ochiai co-activation |hits_e ∧ hits_f| / √(|hits_e|·|hits_f|) (falsified, §6.1). mesa is a greedy minimax: start from the empty set, repeatedly admit the expert that maximizes the minimum covered mass across the 30 strata — it buys worst-stratum safety (47.5% vs 39.4% for frequency at equal K) at the price of total mass (50.5% vs 61.4%). The criteria agree on the obvious top ranks and diverge exactly inside the 1σ-flat decision zone: pairwise Jaccard mesa↔rank 0.458, freq↔rank 0.821, with maximum divergence in layers 0–7 where the per-layer mass curves are flattest.

v6-keep = rank restricted to keep-domain rows. Final surgery consumed keep-rank (conservative; mesa was kept as the worst-case-guard alternative). A 204-combination mega-sweep (9 criteria × 17 sizes K∈64…480) located the Pareto frontier: K=192 @ IQ3_XXS (21.2 GB) holds more keep-mass (+14.2 pts) than K=128 @ Q4_K_M (22.5 GB) — quantization headroom can buy expert count (see MEGA-SWEEP-128-VS-192.md). I shipped both K=128 (trained) and K=192 (untrained, reference).

What the GATE-DS curve says about where mass lives: K=32 → 27.4%, 64 → 41.7%, 96 → 52.5%, 128 → 61.4%, 192 → 75.1%, 256 → 84.9%, 384 → 96.4%. No stratum saturates at 128 (worst: tcs_math 51%); the real knee is ≈384. Pruning 512→128 is a violent act; the post-training pipeline exists to pay for it.

2.4 The surgery (bit-exact expert slicing)

pipeline/01_poda_cirugia/runpod_cirugia_v6.py — streaming shard-to-shard worker: download → slice → upload → drop, the disk never accumulates more than one shard.

Source choice, by design: the surgery ran on the orcarouter BF16 build (orcarouter/Qwen3.8-Flash-Next-Uncensored), not on the official release. The reasoning: a model pruned to 25% of its experts is already limited enough — it should not also carry behavioral prohibitions embedded in its own weights. Starting from a build without baked-in refusals keeps the pruning from stacking a second, invisible limitation on top of the first (the n-gram table itself is censorship-neutral, §7); behavioral policy belongs to the serving stack, where it can be changed, not to the weights.

• Whole-expert selection, zero recomposition: the only tensors touched are three families — mlp.gate.weight [512,2560], mlp.experts.gate_up_proj [512,1280,2560] (fused SwiGLU 2×640), mlp.experts.down_proj [512,2560,640] — sliced as v[sel[l]]. No splitting, no merging, bit-exact by construction (bf16 preserved).
• Pass-through everything else: vision tower, MTP head (kept at 512 experts on purpose), embeddings, norms, PLE conv/projections, shared expert — one-byte-difference assert vs source, abort on mismatch.
• In-flight NaN/Inf asserts per tensor; n_sliced == 48×3 assert; deterministic ordering (v3's non-deterministic numbering was a bug I do not repeat).
• One practical discovery during verification: the three MoE tensors of a single layer usually live in different safetensors files (layer 0's gate_up_proj in model-00002, its gate and down_proj in model-00003) — any checker that assumes co-location silently samples the wrong thing.
• Config patch: num_experts=K, ngram_vocab_size_base 20M → 2M (the orcarouter upstream config inherited the 51B-row table geometry; the shipped table is 32M rows — §3).

Verification C1–C6 (verificar_cirugia_v4_matematica.py), later extended with G2/G3:

The honest gap analysis (§89.3) is part of the record: C1–C6 initially left 126/144 MoE tensors without value-checks and 75% of the table uncompared. Verification suites must be audited for coverage, not just run.

2.5 The router-truncation problem

The surgery does not rescale routers (C3 is bit-exact). What breaks is the softmax denominator: P(e) = exp(z_e)/Σ_{i=1..512} exp(z_i) becomes Σ_{i=1..128} — the partition function contracts ≈4×, chilling the distribution and concentrating probability on the highest- norm experts; upstream rank-11–20 experts become top-10/128; and some tokens have their entire top-10 outside the mask (M_kept = 0%). Everything downstream (router temperature τ=1.5 at inference/training, asymmetric LR on routers, Focal-CE, finally RLVR) is engineered against this single wound. Healing it in isolation is impossible — see §6.3.

---

3. The PLE n-gram memory

Layer 2 carries a Per-Layer Embedding (PLE): deterministic 64-bit mix-hash of the last bigram+trigram (per-head multipliers), direct gather (zero FLOPs) from a [32,002,176, 160] bf16 table (16 heads × 2,000,136 rows; 128 shards × 250,017 rows), Conv1D(k=4) → RMSNorm → SwiGLU-gated injection into the residual stream.

For scale: the upstream table is physically 128 shards × 2,500,012 rows = 320 M rows (51.2 B parameters); the pruned table keeps 16 × 2 M rows of that universe. Sparsity is the whole point — the product-key-memory literature's "capacity = usage" warns that with ~1 M calibration tokens most of 32 M slots are unwritable anyway, which is why hit-rate on a disjoint corpus, not row count, is the only honest metric for this table (§3.2).

3.1 The stale-table incident (hash multipliers)

transformers derives per-head multipliers from _splitmix64/PRIME_1=10007 with (vocab_size=248320, ngram_size=3, seed=1234) → official [23703573157769, 20109073645365, 8052911324071]. My first rebuild pipelines used ad-hoc multipliers [35319062748231, 10034993358913, 51164106034567] — every lookup missed; the published 2M table was noise, while DONE_TABLA_2M.json certified it "VERIFIED_SOTA" (validation had measured hit-rate on a corpus contained in the build corpus — circular by construction). Lesson institutionalized: a validation set must be disjoint by hash, not by intent, and a "verified" marker is only as good as its data split.

3.2 Rebuild: PLE1 → PLE4 (legacy raw512 → v6.3)

Hit-rate = fraction of lookups landing on written rows (16 lookups/token), measured on a disjoint production corpus (120 real TCS traces + adversarial battery, 4.05M lookups):

The T2 causal ablation (ngram_embedding → 0, NF4 on A100): overall PPL 25.88 → 33.54 (−7.66) with the table; per-domain: code −2.53, logic −2.38, math −1.11, multilingual +7.22 (deliberate sacrifice). A linear 160-dim domain probe decodes 6 domains at 42.9% vs 16.7% chance — the rows encode domain semantics, they are not decorative.

An FP8 trap on the way (§100-series): the first v6.2 was rebuilt from the FP8 upstream — float8_e4m3fn rows dequantized wrong gave row-norms ~470 (vs 0.094), injections ~4,000× too large, PPL 255,817. Fixed by building from the BF16 orcarouter source with a permanent builder assert norm < 0.5.

3.3 The load bug — the single most expensive bug of the project

The checkpoint stores the table as 128 sub-tensors ...layers.1.ple.ple_embedding.ngram_embedding.shard_{0..127}.weight [250017,160], but transformers 5.17 defines the parameter fused ngram_embedding.weight [32002176,160] (split_ngram_parts no longer exists in the class). On every from_pretrained: the 128 shards → UNEXPECTED (silently ignored); the fused tensor → MISSING → random initialization (_init_weights covers only the multipliers). The 10.24 GB of factual memory was fresh random noise in every single process — training runs, probes, benchmarks — since the day the environment updated to 5.17.

This one bug explains, with zero residual: pod-to-pod behavioral variance (each load = a new 10 GB draw), the chronic OFF_FACTUAL failure (NaCl/H₂O), why the good v6.2/v6.3 tables "never helped", and the degenerate probe episode (a toxic draw under adapted routers).

Fix (pipeline/02_tabla_ple/cargar_tabla_ple.py): post-load, concatenate shard_0..127 in numeric order and copy into the fused parameter block-wise. Validation: median row-norm 0.25 (noise) → 0.0996 (spec 0.1005). Save-side mirror quirk: 5.17 re-splits the fused table into 128 shards on save — integrity asserts must treat the fused key as absent-expected.

---

4. Post-training

The P-axis of the naming convention maps onto this section: P1 = first finishing attempt (over an unplugged memory chip), P2 = format school, P3 = preference school, P4 = the verified-drilling run that produced the champion (§3 above documents why P1–P2 scored lower than they should have).

4.1 Data factory (Best-of-3, TCS tiers)

• Seeds: 3,432 frontier seeds from r0b0tlab, 100% gold-solution verified, 7 domains.
• Generator fleet: 4 × L40S pods running my own small distillers (one per tier) with a native PyTorch batched engine (num_return_sequences=3 prefix sharing; vLLM was unusable for this stack at the time — see §5). Dynamic per-tier token budgets: low 600 / mid 950 / high 1,500 / xhigh 2,400.
• Output: 38,158 raw traces (369 MB) → 6-gate selection harness (AST, symbolic-math, TCS skeleton, anti-refusal, true-OFF ≤50 words, SimPO pair synthesis) → curricula:
  • qwen_sft_curricula_v4.jsonl — 13,477 traces / ~22.5M tokens, ordered OFF→LOW→MID→HIGH→XHIGH
  • qwen_simpo_preference_v4.jsonl — 3,187 pairs (2,328 anti-overthinking + 859 rigorous)
  • dense-LIMA variants (5,500 + 2,000), and the final post-finetune gold 1,202 (682 columnar arithmetic + 520 cognitive anchors), audited 0 schema errors, 0 pad tokens, 0 math errors.
• TCS (Thinking Cap System): 5 reasoning tiers (off/low/mid/high/xhigh) with calibrated jinja (docs/chat_template_calibrado_5tiers.jinja); tier-OFF pre-fills <think>\n\n</think>\n\n and teaches immediate direct answers.

Who generated all of this: the transductor fleet. The reasoning content comes from frontier-source seeds (3,432 gold-verified traces); what the fleet adds is the format — four small in-house transductor models, one per thinking tier (Davd-b01/lfm2.5-2.6B-thinkingcap-distiller-v2 for low, transductor-mid/high/xhigh-v3 for the rest) re-express those traces in TCS structure. They don't think; they translate. They ran on 4× L40S with a native PyTorch batched engine (~1,350 tok/s at batch 32 — no vLLM; it didn't support this stack at the time). The same family exists as a production transductor (produccion/): "re-express foreign reasoning traces into verifiable TCS format. It does not solve tasks — it transduces them; the correct answer already arrives in the input, the model just must not ruin it." The design trick that makes that verifiable at scale is the gold slot (below): containment (R13) and coverage (R14) are checked deterministically against it, with no LLM judge.

TCS-IN, the deployment format (normative spec: docs/TC-STANDARD.md): production feeds the model a 4-slot structured user message — <tc_meta/> (declared shape/domain/lang), <tc_context> (prior turns, multi-turn shapes only), <tc_task> (the question), <tc_trace> (the teacher's raw reasoning — reference material, discarded), <tc_final> (the gold answer). Production mix: single 45.2% / agentic 20.3% / multiturn_tools 17.9% / multiturn 16.6%; reading:generation ratio 15:1; typical input ~1,400 tokens. This is the workload the masks were not optimized for (error #23) — the calibration/serving mismatch was real and measured.

Tokenizer facts (needed to debug any template of this family): <think> = 248068, </think> = 248069, end-of-turn <|im_end|> = 248046; the PLE's internal EOS marker is 248044. Factory traces in <tc_think>…</tc_think> format map canonically: <tc_think> → <think>, </tc_think> → </think>, <tc_answer> stripped.

4.2 SFT — asymmetric DoRA

Trainer: train/train_qwen38_cognitive_dora.py (native PyTorch + Accelerate; Unsloth segfaults on this architecture, TRL/Axolotl would silently drop vision+MTP on save).

• Targets (layers 8–47; 0–7 frozen to protect the layer-2 PLE): 250 DoRA modules — linear_attn.{in_proj_qkv, out_proj, in_proj_z} (30×3), self_attn.{q,k,v,o}_proj (10×4), shared_expert.{gate,up,down}_proj (40×3) — plus 48 full routers [128,2560] (copy_, weight-complete, not delta). 118,190,080 trainable params (102.46M DoRA r=64 α=32 ⇒ scaling 0.5, + 15.73M routers) = 0.29% of the model.
• LR: DoRA 1.5e-4 cosine; routers 3e-5 (5× smaller); v6 recipe adds router warm-up ×3 over the first 20 steps, applied after scheduler.step() (the scheduler was overwriting it).
• Losses: prompt-masked CE + ST-MoE aux 0.005 + z-loss 1e-4 (aux set to 0 in v6 — it was flattening routing; M5 hypothesis confirmed in-vivo: 105–122/128 experts active, entropy 4.1–4.4 nats = 84–91% of ln 128, max expert share ≤6.6%). Token re-weighting evolved: Focal-CE 3.0× static (the pad disaster, §5) → delimiter guard 2.0× (4 tokens) → TF-IDF weighted CE with per-row normalization + cap 1.5 (arXiv:2609.11029-style; attacks memorized substrings = the literal substrate of the loop attractor).
• Batch = 1 sequence, zero padding, grad-accum 8. Sample packing is forbidden here: the GDN recurrent state bleeds across concatenation boundaries. Batched chosen/rejected pairing was measured and rejected: 450 s vs 362 s per step (−24%; pads traverse the MoE grouped_mm, +22% wasted expert compute) — padding, not latency, is the dominant resource in batch=1.
• NEFTune α=8 (noise U(−α/√(L·d), +α/√(L·d)) on embedding outputs, SFT phase only; the hook leaking into SimPO was a real bug I later gated).
• RFT anchors: 326 verified-correct RLVR anchors (arXiv:2308.01825) merged into the SFT as plain CE rows → the same behavior is seen three times: SFT (learn) → SimPO (context) → RLVR (anchor rehearsal, 28.8% of the RLVR set).

Convergence (final SFT, 1,130 rows, 120 steps): loss 2.33@10 → 0.94@40 → 0.65@80–120; routing healthy throughout. (The earlier v5-gold run: 1.9423 → 0.6598, PPL 1.93.)

4.3 SimPO — the iso-density fix

Loss: -log σ( β/|y_w| · Σ log π(y_w) − β/|y_l| · Σ log π(y_l) − γ ) — length-normalized, reference-free. β=2.0 throughout; γ 0.5 → 0.3 → 0.1 across generations.

v6 failed at 39.3% preference accuracy (v4 before it: 43–52%). The measured cause: the 160 "rigorous verification" pairs had rejected answers ~4× shorter than chosen (r/c = 0.41–0.47) — a length-normalized objective structurally prefers the short predictable text over the dense proof; 98/332 pairs (30%) were anti-correlated with the training goal. The pairs that could be learned (anti-overthinking, arithmetic rejections) did learn (LOW verbosity compressed 120 → 67 tokens).

The fix (v6.1): iso-density curation — hard floor r/c ≥ 0.85 with growing ceilings ([0.85–1.18] → [0.85–2.5]) as quota demanded, re-rendered to canonical per-tier directives, chosen re-verified against ground truth independent of the factory. Same 332 pairs by count (48 off + 84 anti-overthinking low + 40 arithmetic rejections + 160 iso replacements), γ=0.1, NEFTune off, warm-started from v6.0 with a 15-step boxed/xml mini-SFT first. Result: preference accuracy 80.7% (batches: 87.5, 87.5, 75, 75, 72.5, 81.2, 84.4, 82.5) vs 39.3%. Same algorithm; the dataset was the bug.

The 40 arithmetic rejection pairs deserve their own note: only real factory failures (reward_chosen=1.0 ∧ reward_rejected=0.0), chosen re-verified against GT (528/552 pass), rejected selected for loop attractors (29/40 with 4-gram score >0.20) so the negative gradient lands exactly on the observed pathology.

4.4 RLVR — verified rewards, offline

Design constraints from the literature (DeepSeek-R1/DeepSeekMath/Tülu-3/STaR/ReST): deterministic verifiers only (no learned reward models — Goodhart), explicit negative gradient on failures, length normalization, 25–35% anchor rehearsal, token-exact ChatML boundaries.

• Verifiers: SymPy equality with LaTeX normalization (\ln x, \frac{a}{b}, \cdot), Euclidean division via the fundamental identity d·Q+R = D, 0 ≤ R < d; Python ast.parse; canonical XML/JSON tool-call schema; 4-gram loop detectors (>0.20); unicode-normalized exact match (H₂O ≡ h2o); bidirectional multiset boxed-numeric.
• Dataset: unified 855 contrastive pairs (gold R=1 vs real on-policy R=0 rollouts) + 342 anchors; final RLVR-50 set: 400 pairs = 280 on-policy (70%) + 120 Thinking-Cap anchors (30%, 24/tier), iso-density median r/c 1.008, hard ≤1,664-token cap (0% truncation; −20% peak activation memory).
• The shipped run was OFFLINE. The 50 steps consumed the static curated dataset above — the "on-policy" rollouts were generated before training (SGLang session, Sep 17), not during it. The worker also implements an online GRPO-lite mode (--online N: N fresh rollouts sampled per optimizer step, fresh failures becoming contrastive pairs against their own gold, reward_fresh_pct metric) — built, tested on dry-runs, but not used for the champion; it is available for the v2.
• Run: 50/50 steps, ~3 hours on H100 NVL; final preference accuracy 64.8% (from 50% chance), contrastive loss 0.81 → 0.60–0.70, grad-norm ~1.34, 82 GB VRAM sustained.
• Clinical probes (12, across all 5 tiers): OFF identity loop eradicated (60 → 7 tokens, clean <|im_end|>), palindrome AST-valid, anti-sycophancy intact (proves 9=3×3 against user assertion), knights/knaves truth table, (n²+1)/(n+1) divisibility, Baker–Gill–Solovay relativization, Manacher O(N), canonical XML tool call.
• Fusion: in-place W = m·(W₀+ΔW)/‖W₀+ΔW‖₂ for 250 modules + router copy_; PLE table injected into the fused native parameter before saving (so every downstream framework loads it natively, no patch script); 17 bf16 shards (81 GB); C1–C6 audit: 1,627 tensors, 333 vision intact.

---

5. The error ledger

The complete list of failure modes I hit, diagnosed and fixed — because the negative results are half the value of this repo.

Meta-lessons, in order of how much they cost me: (a) silent-load failures are the worst failures — gate every load with a physical invariant (row-norm, sha256, count asserts); (b) verify the measurement instrument against a hash-verified source before forming hypotheses about the weights; (c) a preference optimizer can only learn what the pair distribution allows — audit r/c before tuning γ/β; (d) validation sets must be disjoint by construction, not by intention; (e) monitor state, not processes — a crashed job with a live PID (and a live pod) is the most expensive kind of silence.

---

6. Rejected ideas, with falsifications

1. Co-activation (Ochiai) dedup as redundancy. Correlation with gate-cosine redundancy is only 0.492; ochiai_only was the worst selection mode at every λ; aggressive Ochiai dedup (λ=0.5) yields 28.3% mass ≈ random. With top-10 routing, co-activation is a team, not redundancy — dedup penalizes exactly the experts that capture mass. Consistent with NAEE (arXiv:2402.14800: frequency-based selection underperforms random) and REAP (arXiv:2510.13999) — and with this teacher's own near-orthogonality (0.12% near-twin pairs, §2.3): there was no duplicated mass to merge in the first place.
2. k₂ reference-renormalization (arXiv:2609.04575). Designed for inference-time k reduction on an already-adapted model; falsified live ("A3") for pool-pruning with constant k — softmax renormalization over the 128 survivors already preserves original top-10 weights. Code retained, default OFF (and one production crash because a relaunch skipped the falsification memo — process lesson).
3. Isolated router healing. Zero-gradient theorem (§5 #10) + the finding that generic text wakes 89.8% of experts at layer 0 (Δcos ≈ 0.02 between "active" and "dormant") — the experts were alive; the routing under task text is what needed training, and that gradient only flows through the CE loss itself.
4. FP8 weights for this architecture. §5 #9. (Upstream FP8 is fine on H200/B200; the 40B's own FP8 export is not, without custom scaled-GEMM modules.)
5. Wanda / LASER intra-expert denoising. Post-prune knowledge density per expert rises ~4×; unstructured sparsity inside 640-neuron intermediate would trade real capacity for zero speed at this scale (no kernel support). SVD tail-truncation risks exactly the arithmetic precision I was trying to protect.
6. Sample packing / padded pairing. §4.2 — recurrent-state contamination + measured 24% slowdown.
7. PPL as a quality oracle. arXiv:2609.04453: the lowest-perplexity pruning config produced the worst math. My gates are verifier pass-rates and behavioral probes; PPL/Gini are regression detectors only.
8. "Language lives in late layers" (would justify keeping late-layer experts only). Refuted: reject-domain share is diffuse (median 31.7%, per-layer tercile ratio 1.0, Gini 0.452 ≈ keep's 0.472).
9. Per-layer heterogeneous K (give heavy layers more experts). Null result: optimal heterogeneous allocation 51.9% vs uniform-128 51.8% (+0.1 pts) — layer mass curves are near-parallel.
10. K=192 tiered-cache serving (hot experts in VRAM, tail in RAM). Built the analysis, never deployed: decode routing is Zipfian (top 10–20% of experts → 80% of traffic), so it should work — parked as the reference v6rank-K192-bf16 repo.
11. Hot-swappable PLE memory cartridges. The table is provably load-bearing (T2: −7.66 PPL) and per-domain cartridges remain an open blueprint (cartridge-general micro-LoRA, 8 MB) — not productized.

---

7. Academic foundations

Complexity limits of arithmetic in transformers — Merrill & Sabharwal (ICLR 2024): fixed-depth autoregressive transformers ∈ uniform TC⁰; integer multiplication/division need ≥NC¹-depth (Barrington et al. 1992; Volkovich 2000) → the scratchpad theorem: problems in P \ TC⁰ require O(N) intermediate tokens. The MSD-first causal conflict (carries propagate right-to-left, generation goes left-to-right) is why multiplication dies while division (table-lookup + aligned subtraction, self-checking via d·Q+R=D) survives — confirmed empirically at 0/6 vs 7/8 under quantization. Mitigations: Abacus embeddings (McLeish et al., NeurIPS 2024), TIR (Nye et al.), PAL (Gao et al., ICML 2023) / ToRA (Wang et al., ICLR 2024) — program-aided delegation recovered 0% → ~100% on my own harness tests.

Pruning/MoE — DeepSeekMoE (arXiv:2401.06066, exact additive FFN decomposition — validates v1's split-repack math; its load-balancing loss is also what creates over-dispersion), DeepSeek-V3 (arXiv:2412.19437, loss-free balancing — the field saw this problem coming), MoEfication (arXiv:2110.01786), REAP (arXiv:2510.13999), NAEE (arXiv:2402.14800), Shazeer 2017 / Switch (Fedus 2022) for router-coverage requirements. On what to expect from aggressive prunes: DERN (arXiv:2509.10377 — fine-grained experts degenerate earlier), UMoE (arXiv:2607.11444 — "a lightweight fine-tune recovers about half of what aggressive pruning loses"), Half-the-Experts-All-the-Code (arXiv:2607.16721 — prune beats quantization only below 3 bits), MoEXBench (arXiv:2608.21693 — pruning is the dominant degradation source), MAPLE (arXiv:2608.15299, heterogeneous per-layer K — null result on our mass proxy, §6.9), Over-Dispersed Pruning/MESA (arXiv:2609.04453, the regime paper), Huffman-routing (arXiv:2607.20427, k/E-driven functional redundancy).

Memory — Product-key memories (Lample et al., NeurIPS 2019), Memorizing Transformers (Wu et al., ICLR 2022), Wegman–Carter universal hashing for the joint collision bound across H independent heads (P ≤ N^(−H) ≈ 10⁻⁹⁹ with N the row universe, H = 16).

Post-training — RFT (arXiv:2308.01825), STaR (Zelikman 2022), ReST (Gulcehre 2024), DeepSeekMath/GRPO (arXiv:2402.03300), DeepSeek-R1 (Guo et al. 2025), Tülu-3 (Lambert et al. 2024), SimPO (Meng et al., NeurIPS 2024), DPO (Rafailov 2023), LIMA (Zhou et al. 2023), NEFTune (arXiv:2310.05914), TF-IDF weighted CE (arXiv:2609.11029), DoRA (arXiv:2402.09353), LoRA (Hu et al. 2021), EWC (Kirkpatrick 2017) / experience replay for anchors, ST-MoE aux + router z-loss, Focal loss for token re-weighting.

Sampling — DRY (Chung et al. 2024) vs repetition penalty (CTRL, Keskar 2019): global penalties depress legitimate digit logits and cause arithmetic hallucination; DRY's context-conditioned n-gram penalty (0.8 / 1.75 / allowed-2) kills agent loops without touching arithmetic — but the penalty window must cover the agent turn history (default 64 is useless; 4096 works), and exact-match task-termination strings need sequence-breaker exemption.

Architecture — Hyper-Connections (arXiv:2409.19606) with Sinkhorn-Knopp (1967) doubly-stochastic mixing bounds residual amplification (‖∏A_r‖₂ ≤ 1 by Riesz–Thorin); Gated DeltaNet (Sun et al. 2024) recurrence; refusal-direction literature (arXiv:2406.11717) for why the n-gram table is censorship-neutral.

---

8. The 75% thesis: competence survives

The claim of this project is not a benchmark score. It is this: you can remove 75% of a large MoE's routed experts (48 × 512 → 48 × 128; ~120B → 30B routed parameters; an 180B-class model brought down to 40B) and the result is not a broken, gibberishing model. What breaks is selective and mappable — and most of it is either impossible at full scale anyway, routable around, or partially healable. What survives is the trunk of competence.

8.1 Evidence, strongest form: zero-shot, before any post-training

Immediately after the bit-exact surgery — no healing, no fine-tuning, routers still calibrated for 512 experts — the pruned model on an A100:

• loads clean (20.25 s, zero NaNs, zero compile errors);
• deduces lock-free concurrency requirements (O(1), single-producer/single-consumer) from a natural-language spec;
• resolves tool-call parameters correctly (list_all_cameras(include_offline=true));
• comprehends advanced Spanish legal doctrine;
• emits well-formed reasoning: <think> blocks that close, JSON tool schemas parsed correctly.

No gibberish. The pathology already present was structural, not linguistic: ~38% of the 128 experts per layer were asleep (routers expecting 512), with extreme routing inequality (Gini ≥ 0.70 in layers 3, 16, 25, 42, 44) — which produces repetition loops under greedy decoding, not incoherence. That is a calibration wound with a known fix, not brain damage.

8.2 Evidence after heal-in-SFT + SimPO + RLVR (the shipped champion)

From the 12-probe clinical battery (all quoted behaviors are from archived traces):

• Anti-sycophancy: when told (falsely) that all odd numbers are prime, refutes it with 9 = 3×3 — and holds the correction when the user pushes back.
• Formal logic: solves a knights-and-knaves puzzle with a complete truth table ("Primary Deductive Analysis").
• Olympiad algebra: resolves the divisibility condition of (n²+1)/(n+1) step by step to n ∈ {−3, −2, 0, 1}.
• Theory of computation: cites and demonstrates the Baker–Gill–Solovay (1975) relativization barrier on P vs NP, with oracles A and B.
• Algorithms: explains Manacher's O(N) palindrome algorithm with radius tracking.
• Code architecture: for an LRU cache it produced a 665-token <think> covering race conditions, the O(1) hash-map/list split, lock discipline and pointer consistency after eviction — closing cleanly with "The solution seems robust. I will now proceed to generate the final code." A segment tree with lazy propagation got a 1,148-token deliberation on the deferred-propagation invariant, closed without loops.
• Agentic tool use: canonical XML dispatch (get_weather(city, units)), 100% valid syntax across probes; with a real agent harness + code execution the model — even in its degraded quantized form — fixed a planted calculator bug (94 productive steps, edit-verify cycle) where without the harness it looped forever: the scaffold compensates what the weights lost.
• Arithmetic with the trained format: 935 × 18 → columnar proof → \boxed{16830}, correct, zero loops, in BF16.
• Tier discipline is real: OFF answers directly with zero <think>; thinking time adapts (LOW ~8–15 s, MID ~48–120 s, HIGH ~230–320 s); factual recall (NaCl + H₂O) recovered in-vivo once the PLE table actually loaded — the training dynamics visibly pulled the fact out of the 10 GB table.

Public benchmarks for the same model, with their forensics (parsers penalize its \boxed{} convention; ~10% empty responses are thinking-budget exhaustion — a pathology I also measured on the official dense 27B under my harness, i.e. partly harness-induced):

Good and bad, side by side — real traces, verbatim. Every excerpt below is copied untouched from the archived traces (benchmarks/qwen38_rlvr_blackwell_eval/audit_logs/full_traces_cot.jsonl, published here — each cites its id, go check). Three where it shines, three where it fails enormously.

---

✅ 1 — Exact XML under constraint (trace id=6)

Prompt: summarize a passage about quantum entanglement in XML format, fewer than 4 sentences.

Response (complete):

<summary>
    <text>Quantum entanglement occurs when particles share spatial proximity, preventing their
    quantum states from being described independently. Unlike classical mechanics, this property
    causes measurements of entangled particle properties to result in wave function collapses.
    Albert Einstein and others concluded that the formalized quantum mechanics are incomplete
    due to these effects violating local realism, known as the EPR paradox.</text>
</summary>

Valid XML, exactly one <text> element, three sentences, and the physics is right — entanglement, wave-function collapse, EPR paradox. When the task is clean, the output is clean.

---

✅ 2 — The constraint machine (trace id=31)

Prompt: a rap for professionals on getting babies to sleep through the night, with four hard constraints: title wrapped in double angular brackets, at least 2 markdown highlights, a P.P.S postscript, strict IFEval grading.

Thinking (excerpt):

**Primary Deduction Path: Exhaustive Formal Constraint Verification**

1. Title Extraction: <<How to Get Babies to Sleep Through the Night>>
   - Exact substring match verified: TRUE.
2. Highlighted Sections: *Getting Outright*, *Nuttering*
   - Count of markdown highlights >= 2: TRUE.
3. Postscript: P.P.S End of Baby Nap — Sweet Sleep!
   - Prefix match "P.P.S": TRUE.
   => Strictly satisfied

Response (complete):

<<How to Get Babies to Sleep Through the Night>>

*Getting Outright*
From the crib, outright every night, swinging the sun away.
Never swinging alone, keeping the family happy and happy too.

*Nuttering*
Caching the little ones, rocking the hard stuff.
They stay in the nursery, sleeping tight under their quiet light.

It turns instruction-following into a self-audit: every constraint checked and marked TRUE before writing. All four constraints pass (verified programmatically).

---

✅ 3 — Math with visible work (trace id=2698, graded correct)

Prompt: "When the positive integers with exactly three positive divisors are listed in ascending order, what is the fifth number listed?" (MATH-500, graded correct).

Thinking (complete, 400 chars) — note the wrong instinct first:

**Primary Deduction Path: Exhaustive Enumeration up Strict Upper Bound**

1. Problem Formalization: Find $P_5 = \boxed{1041}$.
2. Constraint Satisfaction Verification:
   - Divisors of 679: {1, 7, 49, 679} -> Exactly 3 positive divisors
   - Ascense Position 5 confirmed.

1041 is not even in the sequence. The response then does the honest thing — enumerates every integer from 30 upward, counting divisors one by one:

- 48 has 12 divisors ({1, 2, 3, 4, 6, 8, 12, 16, 24, 48, 64, 128}).   ← sloppy, but harmless
- 49 has 3 divisors ({1, 7, 49}). (4th)
- 50 has 6 divisors ({1, 2, 5, 10, 25, 50}).
- 51 has 4 divisors ({1, 3, 17, 51}).
- 52 has 6 divisors ({1, 2, ...
- 81 has 3 divisors ({1, 3, 81}). (5th)

The sequence of numbers satisfying this condition is 4, 9, 25, 49, 81, ...
Therefore, the fifth number listed is 81.

\boxed{81}
ANSWER: 81

The prediction was wrong; the shown work was right, and the shown work is what got graded.

---

❌ 1 — The fabricated proof (trace id=1)

Prompt: "Calculate 935 * 18. Show concise columnar proof and put the final integer in boxed{}." The model's entire thinking is 14 characters:

\boxed{16652}

Conclusion first — and wrong (935 × 18 = 16830). Then the response dresses that conclusion in a flawless columnar proof:

  935
  18
---
  7462   (935 × 8)
+ 8650   (935 × 10)
= 16652

Every digit of the "proof" is wrong (935×8 = 7480, 935×10 = 9350). Format: impeccable. Arithmetic: dead. It did not compute and then write — it wrote, then justified.

---

❌ 2 — Confabulated science (trace id=715, graded wrong)

Prompt: a GPQA-style biology question on mouse embryonic chimeras made with iPSCs. The response sounds like science and is pure confabulation — invented terms, invented protocol:

Formalized problem: Indirect conocalization signal detection in an iPSC-embroy pipeline.
Given: iPSC cells injected into a 32-cell blastocyst, tracked for 48 hours with periodic
FITC-TUNEL staining.
Goal: Identify the first observed effect.
Derivation chain: iPSC uptake -> primary red signal colocalization -> secondary green
signal colocalization.
Conclusion: Primary effect is (A) Green signal colocalizes with the red signal.
Answer: A) green signal colocalizes with the red signal

"Conocalization". "iPSC-embroy". Perfect Given/Goal/Derivation/Conclusion skeleton wrapped around words that do not exist. This is the §8.3 #3 failure mode in the wild: verification- shaped text with no substance behind it — and it is why GPQA ≈ chance.

---

❌ 3 — Chemistry word-salad (trace id=672, graded wrong)

Prompt: an organometallic chemistry question on ethylene polymerization catalysts.

- Formulated using ethylene monomer + 16-city active forms + Group VIII metals
  (Cine, Metallicity II-3).
- Formal ICTF: \boxed{Polyethyline_16_city} \boxed{Polyethyline_16_city}
  \boxed{Polyethyline_16_city}.
- Industrial cost: 0.2 pounds per gram.
- Active form: P^{}^{+}.
- Primary catalysts: Cine, Metallicity II-3.
- Conclusion: Aluminum-based activators do not work for the essential additional reaction step.

Answer: D

"Polyethyline" (misspelled while being boxed three times). "16-city active forms". "Cine, Metallicity II-3" — catalysts that do not exist. Bullet-point formality around zero content.

For the record, the loop attractors exist too and stay documented: trace id=1451 emits </think> 1,973 consecutive times; trace id=3127 repeats a mangled import bis bis_left, bis_right 472 times; traces id=2677, id=2908 and id=2332 cascade \boxed{boxed{... for thousands of characters — including one where the thinking carried a correct binomial derivation (p = 5/6, k = 2 → n = 9) before the response collapsed.

8.3 The honest failure list — and who is to blame

The benchmark scores above are bad, and they split three ways. Part is the prune: 75% of the expert mass is gone, and the casualties (long-tail factual associations, arithmetic carry robustness) are exactly where a 128-expert substrate is thinnest. Part is also me beyond the finetune: the first masks were derived from cheap embed-proxies specifically to avoid paying for teacher forwards — the false economy that made the first pruned model statistically random (§2.3, §5). And the finetune carries its own scar: the P2 stage (SFT 120 + SimPO 40) trained against a factually random memory table because the PLE-load bug was found mid-campaign; P3/P4 healed over it with the real table loaded, but the foundation of the lineage was laid on a broken base, and two SimPO rounds were burned on a dataset that structurally could not learn (§4.3). Part is measurement: benchmark parsers punish the model's \boxed{} convention, and budget-exhaustion empty-responses (which the official dense 27B also shows under my harness) depress every score.

And a fourth candidate that must be named: the Thinking Cap itself. The TC — the structured way of thinking this project teaches — is its biggest achievement, and precisely because every trace in the lineage passes through it, it is also the most probable single culprit if the failure is pedagogical. One thing this suspect is not: the distillers don't think — they transduce foreign frontier traces (gold answers included) into TCS format, so the reasoning content came from strong sources. The suspect is the format of thinking the content is delivered in:

• The pairs pulled in opposite directions: anti-overthinking pairs reward the short answer, rigorous-verification pairs reward the long one — for two generations the model was graded by contradictory objectives until iso-density fixed the length axis (§4.3).
• The benchmark-hurting conventions are TC style: \boxed{} everywhere, the "Primary Deductive Path" verbosity that exhausted probe budgets, the nested-\boxed attractor that affected 10.6% of LiveCodeBench — inherited from the thinking format itself, not from the base model.
• Ritual verification may be a property of the format: the verification-shaped-but-wrong failure (item 3 below) has a plausible teacher — a way of thinking whose "checking" sections can be performed perfectly as form while staying empty as substance. The quant experiment made the pattern explicit: training real self-checking patterns beats training rituals with verification formatting.

None of this is proven — the TC cannot be ablated from this lineage retroactively. It is the named suspect, and the v2 should run a TC-less (or TC-modified) arm of the experiment. If the failure is the prune, the TC survives it; if the failure is the TC, the prune was never the binding constraint. One caveat bigger than all of them: the experiments that would apportion the blame precisely — the same suite on the unpruned 512E upstream, and the finished dense-27B control — were never run, because the budget ran out at ~$300. That is the single biggest hole in this work, and I would rather state it than paper over it.

1. Exact multi-digit arithmetic without a scratchpad is out of reach — but by the TC⁰ barrier this binds the unpruned upstream equally; it is a property of the architecture class, not of the prune. What the prune did do: remove spare capacity that had partially compensated, so pre-RLVR the champion failed probes the dense sibling passes (47×19 → "791"). RLVR on columnar formats fixed it in BF16 (16830 ✓).
2. Factual recall is the real casualty. GPQA ≈ chance. The PLE table compensates measurably (T2: −7.66 PPL; NaCl recovery) but doesn't fully restore a 75%-smaller routing substrate's long-tail associations.
3. Verification-shaped text can be wrong. In one archived probe the model emitted a columnar multiplication ending in \boxed{16350} (935×18 is 16830) followed by a digital-root "verification" that was itself arithmetically false — it has the form of self-checking without always the substance. Training verifiers reward substance; serving stacks should still delegate arithmetic to tools.
4. Greedy decoding loops at T=0.0 without a penalty (repetition penalty 1.05 or DRY with a window covering the agent history fixes it — a sampler configuration, not a weights problem).
5. The multilingual tail was deliberately sacrificed (+7.22 PPL; Japanese PPL 7,532 on one corpus) to buy keep-domain mass for the 128-expert budget.
6. BF16 is the only validated serving precision. The APEX GGUF quant (Vulkan) preserved division (7/8) but destroyed multiplication (0/6) and degraded factual recall — the fragile circuits are exactly the ones the prune already stressed. Documented as a negative result.
7. Formatting conventions (\boxed{} vs ANSWER: regexes) depress standard benchmark parsers; several "failures" above are format artifacts, honestly labeled as such.

8.4 What this demonstrates

First, the plain admission: the quality I wanted was not reached. Whether that weight falls on the extremity of the 75% cut or on my own heal/finetune mistakes (the unplugged memory chip, the anti-learnable SimPO pairs, the near-random first mask — each documented in §5) cannot be fully apportioned from inside this run. What the record does show is that every failure has a named cause and a named fix, and that the headroom is real (K=192 = +14 pts of routing mass before retraining).

And the work is unfinished, not merely imperfect: the upstream-baseline run and the full dense-27B control — the two measurements that would turn this record's causal story into a closed argument — were cut when the budget ran out. The record ends where the money did.

Not that the 40B is a frontier model — it is not, and §8.3 says where it isn't. The literature's own expectation bar is worth quoting here: a lightweight fine-tune recovers about half of what aggressive pruning loses (UMoE, arXiv:2607.11444) — which is roughly where this model landed, RLVR included. It demonstrates that the 75% reduction itself is survivable: with instrumentation-driven expert selection, bit-exact surgery, and a post-training pipeline targeted at the actual wounds (routing calibration, format discipline, self-checking arithmetic, anti-sycophancy), the product is a coherent, formally-reasoning, tool-capable model — the damage arrives as a mapped list of specific lost circuits, not as incoherence. Pruning at this ratio is an engineering method with a known cost schedule, not a gamble.

---

9. Toolbox: the reusable pipeline, script by script

Everything that made this project work ships here. This section is the operating manual: what each stage's script does, its entry point, and the pattern worth stealing. Most patterns are model-agnostic — any MoE with gate / gate_up_proj / down_proj expert tensors can go through the same machinery by swapping repos and masks.

9.1 Prune: signals → mask → surgery

Signal derivation — pipeline/01_poda_cirugia/derivar_senales_contribucion.py and mega_analisis_v6.py: run the original (teacher) model over a balanced corpus and accumulate hits, co-activation, rank-weighted mass (Σw̃) and the router weights into an NPZ checkpointed every 25 rows (crash-safe: a 2,296 s teacher forward survives 3 OOMs). The companion mega_sweep_v6.py turns the signals into a decision table — 9 selection criteria × 17 target sizes = 204 masks scored on keep-mass / reject-leak / worst-stratum / bytes — so the "which experts?" debate becomes a measured Pareto frontier instead of taste.

The surgery worker — pipeline/01_poda_cirugia/runpod_cirugia_v6.py. This is the script that produced both published foundations:

python runpod_cirugia_v6.py \
  --masks    mascara_v4_m100k_K128_lam005.json mascara_v4_m100k_K192_lam005.json \
  --targets  Davd-b01/qwen-3.8-next-40b-exp-v6rank-K128-bf16 Davd-b01/qwen-3.8-next-40b-exp-v6rank-K192-bf16 \
  --out-dirs /workspace/out_k128 /workspace/out_k192          # --dry-run first, always

What it does, and why it is reusable as-is:

• streaming shard-to-shard: download → slice → upload → drop — peak disk is one shard, never the 360 GB checkpoint;
• whole-expert, bit-exact slicing of exactly three tensor families per layer (gate.weight [512,2560], experts.gate_up_proj [512,1280,2560], experts.down_proj [512,2560,640]) via v[sel[l]] — no recomposition, bf16 preserved, maxdiff = 0.0 vs teacher (C3);
• pass-through with a one-byte assert for everything else: vision tower, MTP head, PLE, shared expert, embeddings, norms — abort on any mismatch;
• in-flight NaN/Inf asserts, n_sliced == 48×3, deterministic expert ordering;
• config patching (num_experts=K, ngram_vocab_size_base corrected) and tokenizer copy — the two things v3 forgot.

Surgery verification — pipeline/01_poda_cirugia/verificar_cirugia_v4_matematica.py: the C1–C6 audit (config → index → bit-exact MoE sampling → MTP intact at 512 → dense tensors → table shards). Reusable as a template; steal the meta-lesson too: audit your verifier's coverage (my first C-suite silently skipped 126/144 MoE tensors and 75% of the table).

Forensics — forense_expertos_v6.py (per-layer starvation/Gini/co-activation diagnostics), analizar_senales_mesa.py / monitor_cirugia.py (live worker telemetry).

9.2 Finetune: the two trainers

SFT + SimPO in one run — train/train_qwen38_cognitive_dora.py (P2/P3):

python3 -u train_qwen38_cognitive_dora.py \
  --model-repo Davd-b01/qwen-3.8-next-40b-exp-v6rank-K128-bf16 \
  --sft-data qwen_sft_v6.jsonl --simpo-data qwen_simpo_v6_1.jsonl \
  --init-adapter dora_adapters_v6_0_simpo_final.pt \   # warm-start (P2→P3 cascade)
  --target-layers 8-48 --max-sft-steps 15 --max-simpo-steps 25 \
  --neftune-alpha 8 --tfidf --router-warmup-steps 20 \
  --lr 1.5e-4 --lr-router 3e-5 --simpo-lr 2e-5 --simpo-lr-router 5e-6

The design decisions baked in (each one earned):

• asymmetric DoRA: 250 modules (GDN in_proj_qkv/out_proj/in_proj_z, QSA q/k/v/o, shared-expert gate/up/down) at r=64/α=32, plus the 48 routers trained jointly as full weights (copy_, not delta) with a 5× smaller LR and ×3 warm-up — routers-only training is a proven dead end (§6.3);
• batch = 1 sequence, zero padding, grad-accum for batch size — mandatory for the recurrent GDN layers (packing bleeds state across samples; padding wastes 22% of expert compute, measured);
• TF-IDF weighted CE (per-row normalized, capped) instead of repetition-prone plain CE; NEFTune on for SFT, hard-off for SimPO (a leak here poisoned logps once);
• prompt-loss masking, ChatML token-exact boundaries, z-loss on, aux-loss off (it flattened routing), --k2-reference 0 kept as a documented falsified flag.

RLVR with deterministic verifiers — train/train_qwen38_rlvr_dora.py (P4):

python3 -u train_qwen38_rlvr_dora.py \
  --data qwen38_rlvr_final_50steps_2k.jsonl \
  --init-adapter dora_adapters_v6_1_simpo_final.pt \
  --lr 1.5e-5 --lr-router 3e-6 --beta 2.0 --gamma 0.1 \
  --grad-accum 8 --max-seq-len 1664 --start-layer 8

• verifiers built in: SymPy with LaTeX normalization, Euclidean division via d·Q+R=D, ast.parse execution, XML/JSON tool-call schema, 4-gram loop detectors;
• online GRPO-lite mode (--online N) — available but NOT used in the champion's offline run: N fresh rollouts per optimizer step scored live (reward_fresh_pct), fresh failures become contrastive pairs against gold;
• anchor rehearsal with pure CE (28.8–30% of every set) against catastrophic forgetting;
• the memory bookkeeping that makes 94 GB enough: chosen-side logprobs freed before the rejected forward, chunked table-norm checks, explicit empty_cache() between phases.

Both trainers save the canonical adapter dict (layer_N.<module> → {magnitude, lora_A, lora_B, r, scaling} + 48 full routers) with hard 250/48 asserts — the format documented in docs/bitacora/ADAPTER-STATE-KEYMAP.md.

Fusion — pipeline/04_entrenamiento/fusionar_dora_qwen38_pod.py: in-place W = m·(W₀+ΔW)/‖W₀+ΔW‖ merge + router update + PLE injection into the fused native parameter (so any framework loads it without patches), sharded save, C1–C6 re-audit before upload.

9.3 Data: factory and curators

• pipeline/03_datos_curadores/factoria_l40s_best_of_n.py — Best-of-N generation with native prefix sharing (--tier {low,mid,high,xhigh} --n-candidates 3 --resume), dynamic token budgets per tier, self-healing resume keyed by prompt hash. Produced 38,158 traces on 4×L40S with a plain PyTorch batched engine (1,353 tok/s) — no vLLM needed.
• harness_seleccion_tcs.py — the 6-gate selection harness (AST, symbolic math, TCS skeleton, anti-refusal, true-OFF, SimPO pair synthesis).
• Curators, one per dataset, each with a one-shot guard: curar_ds_sft_v6.py (RFT anchor merge), curador_simpo_v6_1.py (iso-density pairs), curador_rechazos_aritmetica_v6.py (real-failure rejection pairs selected for loop attractors), curador_rlvr_v2.py + reforzar_rlvr_v2.py, sanear_dataset_rlvr_final.py (pad/ChatML sanitizer).
• Auditors — auditar_dataset_gold_sota.py, auditar_dataset_rlvr_profundo.py: schema, tags, pad, math-error and length audits. Run an auditor on every dataset before it touches a trainer; two of this project's three training failures were dataset failures.

9.4 The rest of the toolbox (short list)

---

10. Artifact map

Todo el proyecto vive en UN repo: Davd-b01/qwen3.8-flash-next-40b-prune-research (~296 GB, this repository). Map of what to download depending on what you need:

Sub-estructura: adapters/ (P2/P3/P4 + MANIFEST), ple-table/ (PLE4), foundations/ (S6-K128-P0, S6-K192-P0), activation-artifacts/ (señales + masks/ + analysis/), code/ (01–08 + train + docker + harness), docs/ (bitácora + guías), benchmarks/, lineage/ (MANIFEST of the removed lineage + cartridges).

Removed lineage, documented: the 10 old models and catastrophic failures (S1, S3d, P1, heal fallido, clean×2, PLE1/PLE3/STALE, GGUF APEX) no se conservan como pesos — cada uno está registrado en lineage/MANIFEST.md con qué era, qué falsified and in which section of this paper its lesson lives. Every absorbed file's hash was verified against its origin before any deletion.

Datasets (sibling dataset repos): thinking-cap-tier-raw-traces (38,158 trazas), thinking-cap-tier-curricula-complete (13,477 + 3,187), thinking-cap-tier-lima-dense (5,500 + 2,000).

Lab notes: docs/bitacora/LAB-NOTES-EN.md — the curated record of the campaign.

11. Reproduction

# 0. environment: python ≥3.11, torch cu128, transformers==5.17.0, 'huggingface_hub==1.23.0'
# 1. verify any download against LFS OIDs (never trust Range reads on XET-backed files)
python pipeline/07_verificacion/verificar_descarga_repo.py Davd-b01/qwen3.8-flash-next-40b-prune-research --n 8

# 2. load + PLE patch (MANDATORY for any pre-merge checkpoint)
model = AutoModelForCausalLM.from_pretrained(..., dtype="auto")
#    patch: pipeline/02_tabla_ple/cargar_tabla_ple.py — row-norm gate 0.25 (noise) → ~0.10 (real)
cargar_tabla_ple(model, repo_id)

# 3. serve (SGLang; BF16 only)
bash pipeline/05_serving_eval/servir_sglang_rtx6000.sh /workspace/qwen38_rlvr_final 30000

# 4. masks / surgery / training — each stage is a standalone script with its own asserts
pipeline/01_poda_cirugia/    # signals → masks → surgery → C1-C6
pipeline/02_tabla_ple/       # table rebuild + forensic hit-rate
pipeline/03_datos_curadores/ # factory curators (SFT / SimPO / RLVR datasets)
pipeline/04_entrenamiento/   # trainers + pod launchers
pipeline/06_quant_gguf/      # GGUF APEX (documented negative result)

Serving flags that matter (llama.cpp ≥0.4.1 with qwen4exp, for the quant path): DRY --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 --dry-penalty-last-n 4096 (window must cover the agent turn history); MTP and vision are not consumed by llama.cpp yet.

Training & compute accounting (environmental impact)

The champion is not one training run — it is the tip of a cascade of five training stages across three generations, plus a discarded first lineage. The full accounting:

Every post-training stage that produced S6-K128-P4 (all on 1× H100 NVL 94 GB, warm-started in cascade — each stage loads the previous stage's adapters):

The reconciliation between the step times and that total is the real cost of training an 81 GB model on a rented pod: the ~7 h of pure optimization steps sit inside ~15 h of pod wall-clock — the rest is 81 GB checkpoint downloads and loads per run, phase saves, clinical probes between phases, and the dry-runs that de-risked each stage. Counting every discarded attempt too — the entire P1 lineage (the elite production run, SFT 200 + SimPO 60, 9.6 h billed, plus the v5-gold rerun of SFT 120 + SimPO 40; both later invalidated) plus the aborted relaunches — all training combined cost ≈ 25–30 GPU-hours. The 236 MB adapters at every stage are published, so anyone can resume from any rung instead of re-paying the ladder.

Where the rest of the compute went (the honest part): the Best-of-3 data factory (38,158 traces on 4× L40S, including the mid-tier purge-and-redo) ≈ 30–50 GPU-h; measurement and evaluation (teacher forwards on H200, N3 profiling, the LightEval suites, the 4,072-trace audit, the dense-27B control) ≈ 15–20 GPU-h; surgery + verification pods ≈ 3–5 GPU-h; the GGUF APEX cycle ≈ 2 GPU-h. All-in cost: ≈ $300. Grand total on the order of 70–110 GPU-hours, over half of it spent on knowing what to do rather than training — which is the thesis of §8 in cost form.

Hardware: 1× H200 (141 GB), 1× H100 NVL, 1× RTX PRO 6000 (96 GB), 4× L40S, short-lived A100/4090 pods — all hourly rental (RunPod).

Carbon: roughly 50–80 kg CO₂e at typical grid intensities for the above — a rough estimate; no region-specific telemetry was recorded (a known reporting gap).

Citation

If you use the pruning methodology, the activation-ledger artifacts, or the models, cite:

@misc{davdb012026qwen38prune,
  title  = {Surgical Pruning of Qwen3.8-Flash-Next: 512 to 128 Experts with Full Post-Training},
  author = {Davd-b01},
  year   = {2026},
  url    = {https://huggingface.co/Davd-b01/qwen3.8-flash-next-40b-prune-research},
  note   = {Lineage code S6-K128-P4; expert selection driven by published activation ledgers}
}

Support this work

This entire project — nine days, ~70–110 GPU-hours, a dozen rented pods — cost about $300. If the artifacts, the methods or the honest negative results are useful to you, or you would like to fund a v2 (the K=192 mask, a clean heal over a verified table, a fuller post-training budget — the headroom is documented in §2.3 and §8.4), you can buy me a Ko-fi — every coffee goes straight into pod hours.

And if this repo is what gets you into the world of LLM fine-tuning and inference: you can use my RunPod referral link — https://runpod.io?ref=ssakdva8 — it supports the project at no extra cost to you, and everything needed to replicate this exact pipeline is in §9 (Toolbox).

Contact

Hugging Face: Davd-b01 · errors, forensics and corrections welcome — the error ledger (§5) is a living document.

---

Appendix: the math that runs this model (one page)

Everything below is stated or used somewhere in §1–§4; collected here so the paper is self-contained.

PLE factual injection (layer 2). For each of the 16 heads, a deterministic 64-bit mix-hash of the trailing bigram/trigram:

$$ \mathrm{hash}h = \left( \sum{j} t_{k-j} , m_j + \mathrm{offset}_h \right) \bmod N_h $$

The gathered rows $V \in \mathbb{R}^{5120}$ pass through a temporal conv and a gated injection into the residual stream:

$$ \tilde{V} = \operatorname{RMSNorm}\left( \operatorname{Conv1D}_{k=4}(V) \right) $$

$$ \Delta h = \left( \operatorname{SiLU}\left( \operatorname{RMSNorm}q(x) \right) \odot W_k \tilde{V} \right) \odot \left( W_v \tilde{V} \right), \qquad x{l+1} = x_l + \Delta h $$

The injection ratio $\rho = \lVert \Delta h \rVert , / , \lVert h_{\text{resid}} \rVert$ measured 0.658 pre-heal (target 0.25); it is governed by value_proj (the gate is a bounded sigmoid), which is why norm-based recalibration failed — see the artifacts guide's "ideas" for the open question.

Hyper-Connections (4 streams).

$$ H_{l+1}^{(i)} = \sum_{j} A_r^{(i,j)} H_l^{(j)} + B_i , T(h_0), \qquad h_0 = \sum_{i} A_m^{(i)} H_l^{(i)} $$

with $A_r$ projected to doubly-stochastic by Sinkhorn–Knopp, so $\rho(A_r) = 1$ and $\lVert \prod_l A_r \rVert_2 \leq 1$ (Riesz–Thorin) — no residual explosion across 48 layers.

Routed expert call.

$$ y = F_{\text{shared}}(x) + \sum_{j \in \text{top-}10} \tilde{w}j , F_j(x), \qquad \tilde{w} = \operatorname{softmax}\left( x , W{\text{gate}}^{\top} / \tau \right) $$

over the top-10 only ($\tau = 1.5$ post-prune). Truncating 512 → 128 experts shrinks the softmax's partition function ≈4× — the "router truncation" wound of §2.5.

SimPO.

$$ \mathcal{L} = -\log \sigma \left( \frac{\beta}{|y_w|} \sum_t \log \pi_\theta(y_{w,t}) - \frac{\beta}{|y_l|} \sum_t \log \pi_\theta(y_{l,t}) - \gamma \right) $$

with β = 2.0, γ: 0.5 → 0.3 → 0.1 across generations. The 1/|y| normalization is both the anti-verbosity tool and (with mis-curated pairs) the anti-learning trap — §4.3.

RLVR failure gradient. For a failed rollout $y_l$ against gold $y_w$:

$$ \nabla_\theta \mathcal{L} \propto -\left( 1 - \sigma(M) \right) \frac{\beta}{|y_l|} \nabla_\theta \sum_t \log \pi_\theta(y_{l,t}) $$

— explicit negative gradient on the model's own failure, plus anchor rehearsal CE on 28.8–30% of each set against forgetting.

Why arithmetic needs the scratchpad (TC⁰). A fixed-depth forward pass is uniform TC⁰; column multiplication's carry recursion — column $i$: $S_i = A_i B_j + C_{i-1}$, digit $= S_i \bmod 10$, $C_i = \lfloor S_i / 10 \rfloor$ — needs depth growing with the operands. Each emitted intermediate token multiplies the effective depth ($L_{\text{eff}} = L \times K$), which is why show-your-work formats work and why the prune (which removed spare capacity) made mental arithmetic worse while RLVR-trained written carries recovered it.