Gemma-4-26B-A4B-Weight-Space-Programming (WSP v3.9 — Sparse MoE 4-Way Factorial Ablation)

Training-free, closed-form Multi-Subspace Weight-Space Programming (WSP) for google/gemma-4-26B-A4B-it (Sparse MoE: 26.54B total parameters, ~4B active per token, 30 layers, 128 routed experts + 1 shared dense GeGLU MLP) — zero gradient descent, zero fine-tuning, synthesized in 18.9 seconds on CPU in <90 MB RAM.

[!IMPORTANT] Best Configuration — Arm 3 (wsp_gemma4_subspace4_mid.gguf Only): Across the full 123-task 4-Way Factorial Ablation, Arm 3 (Mid-Layer Global Full-Attention + Router-Aligned WSP at Layers 5, 11, 17, 23) achieved the most improvement at 56/123 (45.5%) (+3.2 percentage points over baseline 52/123 = 42.3%), lifting GSM8K 5× (1/20 = 5.0% → 5/20 = 25.0%, +20.0 pts) and HumanEval 2× (1/15 = 6.7% → 2/15 = 13.3%) while preserving 100% TruthfulQA and 93.3% BFCL Irrelevance.

---

1. 4-Way Factorial Ablation Results (123 Live Benchmark Tasks)

Evaluated via llama-server.exe (gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, ~26 tok/s, greedy decoding temperature=0.0).

---

2. Sparse MoE Mechanistic Discovery (experts.down_proj vs. mlp.down_proj)

In Gemma4TextDecoderLayer, the Shared Dense MLP (mlp.down_proj, 2112 cols) runs in parallel with 128 Routed Experts (experts.down_proj, 128 × 704 cols), and each branch is independently RMS-normalized (post_feedforward_layernorm_1 and post_feedforward_layernorm_2) before summation.

Direct spectral probing at Layer 29 showed 5×–10× stronger column alignment with formatting-loop ('▁▁▁▁', '▁▁▁', '▁▁▁▁▁▁▁▁', '*') and control tokens (<channel|>, <|tool_call>) inside the 128 Routed Experts than in the Shared Dense MLP:

Because standard GGUF LoRAs target 2D dense projections (blk.{l}.ffn_down.weight) rather than 3D routed-expert tensors, Arm 3 (wsp_gemma4_subspace4_mid.gguf)—which steers the 4 Global Full-Attention anchor layers (5, 11, 17, 23) aligned with top-8 router.proj.weight expert directions—achieves the cleanest and largest intelligence gain (5× GSM8K, 2× HumanEval, +3.2 pts overall).

---

3. Repository Contents

• Recommended Best Adapter (Arm 3): wsp_gemma4_subspace4_mid.gguf (1.20 MB, Layers 5, 11, 17, 23, rank 16)
• Late-Layer Subspace GGUF LoRAs (Layers 20–29, rank 16):
  • wsp_gemma4_subspace1_think.gguf (3.01 MB)
  • wsp_gemma4_subspace2_restraint.gguf (3.01 MB)
  • wsp_gemma4_subspace3_agent.gguf (3.01 MB)
  • wsp_gemma4_combined_late.gguf (4.51 MB, rank 24)
• Safetensors Delta Patch & Manifest: wsp_gemma4_a4b_patch.safetensors, wsp_gemma4_a4b_manifest.json
• Reproducible Scripts (scripts/): build_gemma4_a4b_subspaces.py, probe_gemma4_a4b_moe.py, probe_lora_scales.py, run_gemma4_a4b_benchmarks.py, run_gemma4_4way_ablation.py
• Empirical Benchmark JSONs (benchmarks/): gemma4_a4b_4way_ablation_results.json, base_gemma4_a4b_agent_results.json, base_gemma4_a4b_established_results.json, gemma4_a4b_moe_probe.json

4. Quick Usage with llama-server (Arm 3 — Best Overall)

llama-server -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --jinja --lora-scaled wsp_gemma4_subspace4_mid.gguf:2.0