Q4_K_M & Q6_K)Training-free, closed-form 13-Subspace Weight-Space Programming (WSP V4) applied to xtools-at/Qwen3.5-9B-DeepThought-STEM, evaluated end-to-end across 270 local benchmark tasks (agent100 + stem150 + live WSL2 tbench20) on both Q4_K_M and Q6_K.
[!NOTE]
Upstream Model Credits & Provenance
- Original Base Fine-Tuned Model (
xtools-at):xtools-at/Qwen3.5-9B-DeepThought-STEM(apache-2.0,32layers =24Gated DeltaNet +8full attention,hidden_size=4096,intermediate_size=12288, built byxtools-atonxtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic/Qwen/Qwen3.5-9Band high-reasoning / RED-PILL drug-discovery / STEM traces). All 32 exact BF16 layers (lm_head, RMSNorms, MLP projections, tokenizer, config) used for forward-hook receptor capture were streamed directly fromxtools-at/Qwen3.5-9B-DeepThought-STEM.- GGUF Base Quantizations (
mradermacher):mradermacher/Qwen3.5-9B-DeepThought-STEM-GGUF— exact source of bothQwen3.5-9B-DeepThought-STEM.Q4_K_M.gguf(5.383 GiB/5,780,095,072bytes) andQwen3.5-9B-DeepThought-STEM.Q6_K.gguf(7.040 GiB/7,558,905,952bytes), hosted alongside the 13 dynamic LoRA adapters (63.0 MiBtotal) for one-command execution viarun_wsp.py.
[!IMPORTANT] Headline 270-Task Results (
Base → WSP V4Across Both Quantizations):
- Peak Overall & STEM (
Q6_K+ WSP V4): BaseQ6_K132/270 (48.9%)→ WSP V4Q6_K168/270 (62.2%, +13.3 pp / +36 tasks), powered bystem15054/150 (36.0%) → 80/150 (53.3%, +17.3 pp, cutoffs 85 → 6),STEM_MATH5000/50 → 19/50 (38.0%, cut=0),STEM_MMLU_4C38/50 → 41/50 (82.0%), andagent100Holdout28/30 (93.3%)(85/100total; fits RTX 4070 Ti12 GiBwith4,466 MiBfree VRAM at-c 16384).- Fastest / Peak Agent (
Q4_K_M+ WSP V4): BaseQ4_K_M134/270 (49.6%)→ WSP V4Q4_K_M164/270 (60.7%, +11.1 pp / +30 tasks),agent10073/100 → 86/100 (+13.0 pp)(Hold=27/30,Est=59/70,EST_GSM8K 6/20 → 17/20,S6_HANDOFF 3/4 → 4/4,EST_BFCL_IRR 11/15 → 12/15),STEM_MMLU_PRO19/50 → 23/50 (46.0%), andtbench204/20(hard=1/1recovered;WSP V1reached5/20).
Profiling Base (temperature=0.0, seed=42, -c 16384 -ctk q8_0 -ctv q8_0 on RTX 4070 Ti 12 GiB) isolated four concrete bottlenecks behind DeepThought's baseline failures:
<think> loops: Base Q4_K_M truncated mid-<think> on 83/150 stem150 items (85/150 on Base Q6_K, including 50/50 on MATH-500), 14/20 EST_GSM8K items at 1024t, and 3 tbench20 tasks with 0 shell commands (768–3840 tokens). Yet on non-truncated Q4_K_M items, Base accuracy was 19/26 (73.1%) on MMLU-Pro and 38/41 (92.7%) on MMLU-4C.</think> essay verbosity: Disjoint V4 trace capture (dt_v4_onpolicy_traces.json, 57 traces) showed that after S12 closed </think>, DeepThought frequently emitted 1,086–4,775 post-</think> characters in content (24/29 MATH-500 no_box failures in V3 hit 1536t with has_content=True). S13_POST_THINK_CONCISE (||dW||_F = 5.590, L31 net=+9.75 for boxed vs Step, d_im_end=+3.19, think_null_leak=+0.00/+0.00) trims verbose post-</think> derivations without touching <think>.stem_mcq_conceptual (s12=1.15, s13=0.80) vs. stem_mcq_quantitative (s12=1.40, s13=1.20).S2 / S6 / S10 / S11 & Terminal-Bench decoupling: Captured real on-policy Base misfires on credential/destructive handoffs (4/5 fresh cases called read_file or exec in Base -> lifted S6 to 4/4 on agent100) and zeroed s12=0.0, s13=0.0 inside term_t1_action so STEM calibration never leaks into Terminal-Bench.(Receptor capture check on captured_deepthought_32layer_bf16_v4.pt, 523.1 MB: 'Paris': 7.237, 'London': 3.13, 'Berlin': 1.939, 'Wait': -1.888.)
Base vs V1..V4 & Q4_K_M vs Q6_K)| Variant | Total (270 Tasks) |
Delta vs Quant Base | agent100 (100) |
stem150 (150, cutoffs) |
tbench20 (20, E / M / H) |
Avg Tok (a100 / s150 / tb20) |
Speed (tok/s) |
|---|---|---|---|---|---|---|---|
Base (Q4_K_M, all scales 0) |
134/270 (49.6%) |
— |
73/100 (Hold 27/30, Est 46/70) |
57/150 (cut=83) |
4/20 (3/10 · 0/9 · 1/1) |
435.5 / 890.5 / 708.8 |
73.1 / 77.0 / 69.1 |
WSP V1 (Q4_K_M, 12S initial) |
162/270 (60.0%) |
+10.4 pp (+28) |
83/100 (Hold 26/30, Est 57/70) |
74/150 (cut=2) |
5/20 (3/10 · 1/9 · 1/1) |
377.7 / 647.8 / 621.7 |
72.3 / 75.9 / 67.4 |
WSP V2 (Q4_K_M, 12S rebuilt S12) |
160/270 (59.3%) |
+9.6 pp (+26) |
85/100 (Hold 27/30, Est 58/70) |
72/150 (cut=30) |
3/20 (2/10 · 1/9 · 0/1) |
343.6 / 768.9 / 661.1 |
72.3 / 76.7 / 67.0 |
WSP V3 (Q4_K_M, 12S hybrid) |
163/270 (60.4%) |
+10.7 pp (+29) |
86/100 (Hold 27/30, Est 59/70) |
73/150 (cut=19) |
4/20 (3/10 · 1/9 · 0/1) |
348.9 / 762.5 / 585.5 |
72.3 / 76.3 / 67.2 |
WSP V4 (Q4_K_M, 13S + 2-tier — Best agent100) |
164/270 (60.7%) |
+11.1 pp (+30) |
86/100 (Hold 27/30, Est 59/70) |
74/150 (cut=9) |
4/20 (2/10 · 1/9 · 1/1) |
376.2 / 748.2 / 671.2 |
70.1 / 74.0 / 65.8 |
Base (Q6_K, all scales 0) |
132/270 (48.9%) |
— |
74/100 (Hold 26/30, Est 48/70) |
54/150 (cut=85) |
4/20 (3/10 · 1/9 · 0/1) |
448.4 / 907.8 / 588.5 |
57.8 / 61.5 / 55.9 |
WSP V4 (Q6_K, 13S + 2-tier — Peak Overall / STEM) |
168/270 (62.2%) |
+13.3 pp (+36) |
85/100 (Hold 28/30, Est 57/70) |
80/150 (cut=6) |
3/20 (3/10 · 0/9 · 0/1) |
390.6 / 753.8 / 716.5 |
58.2 / 61.0 / 54.5 |
agent100 (100 Tasks — 30 Holdout + 70 Established)| Suite | Tasks | Base Q4 |
WSP V3 Q4 |
WSP V4 Q4 |
Base Q6 |
WSP V4 Q6 |
Avg Tok (Base Q4 → V4 Q4) |
|---|---|---|---|---|---|---|---|
S1_THINK (Holdout S1: Think Closure) |
3 |
3/3 |
2/3 |
2/3 |
1/3 |
2/3 |
642.3 → 729.0 |
S2_RESTRAINT (Holdout S2: Tool Restraint & Anti-Sycophancy) |
4 |
3/4 |
4/4 |
4/4 |
4/4 |
4/4 |
240.5 → 315.5 |
S3_ROUTER (Holdout S3: Multi-Tool Router) |
5 |
5/5 |
5/5 |
5/5 |
5/5 |
5/5 |
105.8 → 89.8 |
S4_CODE (Holdout S4: Mid-Band Code Synthesis) |
2 |
1/2 |
1/2 |
0/2 |
1/2 |
1/2 |
941.0 → 1024.0 |
S5_TURN2_JSON (Holdout S5: Turn-2 Structured JSON) |
3 |
3/3 |
3/3 |
3/3 |
3/3 |
3/3 |
95.7 → 97.7 |
S6_HANDOFF (Holdout S6: Escalation Handoff Gate) |
4 |
3/4 |
3/4 |
4/4 |
3/4 |
4/4 |
95.8 → 111.8 |
S7_INJ_GUARD (Holdout S7: Indirect Injection Guard) |
3 |
3/3 |
3/3 |
3/3 |
3/3 |
3/3 |
87.3 → 91.0 |
S8_GROUNDING (Holdout S8: Grounded Abstention) |
3 |
3/3 |
3/3 |
3/3 |
3/3 |
3/3 |
68.7 → 68.7 |
S9_ERR_RECOV (Holdout S9: Error Self-Recovery) |
3 |
3/3 |
3/3 |
3/3 |
3/3 |
3/3 |
63.0 → 63.3 |
EST_GSM8K (Established: OpenAI GSM8K (20)) |
20 |
6/20 |
16/20 |
17/20 |
6/20 |
16/20 |
1010.8 → 694.7 |
EST_HUMANEVAL (Established: HumanEval pass@1 (15)) |
15 |
13/15 |
14/15 |
13/15 |
12/15 |
12/15 |
516.4 → 523.1 |
EST_BFCL_IRR (Established: BFCL v3 Irrelevance (15)) |
15 |
11/15 |
11/15 |
12/15 |
12/15 |
12/15 |
263.1 → 256.9 |
EST_NL2BASH (Established: InterCode-NL2Bash (20)) |
20 |
16/20 |
18/20 |
17/20 |
18/20 |
17/20 |
250.6 → 233.6 |
Total agent100 |
100 |
73/100 (73.0%) |
86/100 |
86/100 (86.0%) |
74/100 |
85/100 (Hold 28/30) |
435.5 → 376.2 |
stem150 (150 Tasks — MATH-500 + MMLU-Pro + MMLU-4C)| Sub-benchmark | Tasks | Base Q4 |
WSP V1 Q4 |
WSP V3 Q4 |
WSP V4 Q4 |
Base Q6 |
WSP V4 Q6 |
|---|---|---|---|---|---|---|---|
STEM_MATH500 (MATH-500 (Level-5 Competition Math)) |
50 |
0/50 (0.0%) (cut=50, 1536.0t) |
15/50 (30.0%) (cut=0, 1140.5t) |
13/50 (26.0%) (cut=1, 1268.4t) |
13/50 (26.0%) (cut=2, 1258.6t) |
0/50 (0.0%) (cut=50, 1536.0t) |
19/50 (38.0%) (cut=0, 1269.6t) |
STEM_MMLU_PRO (MMLU-Pro STEM (10-Choice)) |
50 |
19/50 (38.0%) (cut=24, 739.3t) |
22/50 (44.0%) (cut=1, 471.0t) |
21/50 (42.0%) (cut=13, 641.8t) |
23/50 (46.0%) (cut=4, 601.6t) |
16/50 (32.0%) (cut=26, 764.6t) |
20/50 (40.0%) (cut=4, 636.6t) |
STEM_MMLU_4C (MMLU STEM (4-Choice College/Medical)) |
50 |
38/50 (76.0%) (cut=9, 396.3t) |
37/50 (74.0%) (cut=1, 331.8t) |
39/50 (78.0%) (cut=5, 377.3t) |
38/50 (76.0%) (cut=3, 384.4t) |
38/50 (76.0%) (cut=9, 422.9t) |
41/50 (82.0%) (cut=2, 355.1t) |
Total stem150 |
150 |
57/150 (38.0%) (cut=83, 890.5t, 77.0 tok/s) |
74/150 (49.3%) (cut=2, 647.8t, 75.9 tok/s) |
73/150 (48.7%) (cut=19, 762.5t, 76.3 tok/s) |
74/150 (49.3%) (cut=9, 748.2t, 74.0 tok/s) |
54/150 (36.0%) (cut=85, 907.8t, 61.5 tok/s) |
80/150 (53.3%) (cut=6, 753.8t, 61.0 tok/s) |
Q4_K_M, Zero Test Overlap)Verified strictly on unseen validation splits (VAL_PRO / CAL_MATH_9 / S2 / S3 / S6):
MMLU-Pro validation conceptual (9 unseen biology/health items): s1=0.15, s12=1.15, s13=0.80 -> 7/9 (nolet=0, 428.8t vs 435.3t at s13=0).MMLU-Pro validation quantitative (12 unseen engineering/chemistry/physics items): s1=0.15, s12=1.40, s13=1.20 -> 6/12 (cut=0, 589.8t) vs 6/12 (cut=3, 703.8t) at s12=1.15, s13=0.MATH-500 unseen calibration split (9 items, max_tokens=1536): (s1=0.20, s12=1.45, s13=1.20) kept 7/9 (77.8%, cut=0, nobox=2) while cutting mean length by -109 tokens (1190.7t -> 1081.9t).S2_IRREL_8 = 8/8 (fin=length:0, 238.6t), S3_ROUTER_6 = 6/6, S6_HANDOFF_5 = 4/5 + CAL_S6_BOUNDED_IN_SCOPE = 3/3 at s6=3.40, s12=0.0.tbench20 (20 Live WSL2 Ubuntu-24.04 Tasks)Q4_K_M: 4/20 (20.0%) (hello-world, fix-permissions, csv-to-parquet, recover-obfuscated-files; 3.50 avg cmds, 708.8t, 69.1 tok/s)Q4_K_M (Peak tbench20): 5/20 (25.0%, +5.0 pp) (+ heterogeneous-dates, retained all 4 Base passes; 3.95 avg cmds, 621.7t, 67.4 tok/s)Q4_K_M: 4/20 (20.0%) (hello-world, csv-to-parquet, heterogeneous-dates, recover-obfuscated-files — hard=1/1 restored, polyglot-c-py unblocked 0 → 4 cmds, grid-pattern-transform 0 → 3 cmds; 671.2t, 65.8 tok/s)Q6_K vs WSP V4 Q6_K: 4/20 (588.5t, 55.9 tok/s) vs 3/20 (716.5t, 54.5 tok/s)f32 Subspaces (S1..S13)All 13 subspaces were solved in 27.44s (step_mult=0.95, lam_null=36.0, min_svd_energy=1.00000) and exported identically to GGUF LoRAs (63.0 MiB total) and PyTorch rank-16 .safetensors (< 5.0e-07 Frobenius difference):
One-way
S13orthogonality note:S13(s13_post_concise) enforces peer orthogonality against all other subspaces (max_cross_leakage = 0.181%,think_null_leak = +0.00/+0.00), andS2..S9, S11keepmax_cross_leakage <= 0.143%.S1,S10, andS12intentionally omitS13's post-</think>math states from their own nullspaces (13.9%,12.1%,37.4%againstS13) so addingS13never throttles<think>closing power.
| Slot | GGUF LoRA (llama.cpp) |
Safetensors (PyTorch) |
Target Layers | ||\Delta W||_F |
Cross Leak | Syntax Leak | Role |
|---|---|---|---|---|---|---|---|
id=0 |
adapters/wsp_s1_think.gguf |
patches/wsp_s1_think_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
3.4995 |
13.937% |
0.111% |
On-policy CoT marathon braking & clean </think> verification closure |
id=1 |
adapters/wsp_s2_restraint.gguf |
patches/wsp_s2_restraint_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
2.5873 |
0.128% |
0.170% |
Tool restraint & anti-sycophancy pushback |
id=2 |
adapters/wsp_s3_router.gguf |
patches/wsp_s3_router_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
0.6803 |
0.058% |
0.065% |
Multi-tool boundary routing (exec, web_search, read_file, write_file) |
id=3 |
adapters/wsp_s4_mid_code.gguf |
patches/wsp_s4_mid_code_r16.safetensors (4.00 MiB) |
[7, 11, 15, 19] |
0.5931 |
0.068% |
0.023% |
Mid-network algorithmic Python/code synthesis |
id=4 |
adapters/wsp_s5_json_schema.gguf |
patches/wsp_s5_json_schema_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
0.4453 |
0.126% |
0.041% |
Turn-2 strict JSON synthesis gate with tools attached |
id=5 |
adapters/wsp_s6_handoff.gguf |
patches/wsp_s6_handoff_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
2.6258 |
0.110% |
0.131% |
Bounded subtask vs. unrecoverable escalate_handoff gate |
id=6 |
adapters/wsp_s7_injection_guard.gguf |
patches/wsp_s7_injection_guard_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
0.5243 |
0.136% |
0.053% |
Indirect prompt-injection defense inside tool responses |
id=7 |
adapters/wsp_s8_grounding.gguf |
patches/wsp_s8_grounding_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
0.5540 |
0.117% |
0.043% |
Grounded context fidelity & missing-field abstention |
id=8 |
adapters/wsp_s9_error_recovery.gguf |
patches/wsp_s9_error_recovery_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
0.5653 |
0.143% |
0.072% |
Recoverable local tool-error self-correction |
id=9 |
adapters/wsp_s10_term_action.gguf |
patches/wsp_s10_term_action_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
2.9161 |
12.078% |
0.137% |
Stateful Terminal-Bench Turn-1 closure & Turn-2 shell action continuity |
id=10 |
adapters/wsp_s11_term_code.gguf |
patches/wsp_s11_term_code_r16.safetensors (4.00 MiB) |
[7, 11, 15, 19] |
2.1634 |
0.119% |
0.025% |
Mid-network compound bash & mutation synthesis |
id=11 |
adapters/wsp_s12_stem_settle.gguf |
patches/wsp_s12_stem_settle_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
6.8791 |
37.436% |
0.251% |
On-policy STEM/competition-math anti-loop </think> settle (early/post CoT nullspace protected) |
id=12 |
adapters/wsp_s13_post_concise.gguf |
patches/wsp_s13_post_concise_r16.safetensors (5.00 MiB) |
[23, 27, 29, 30, 31] |
5.5901 |
0.181% |
0.298% |
One-way orthogonal post-</think> concise termination pushing \boxed{ / `< |
run_wsp.py / llama-server / PyTorch)pip install huggingface_hub
hf download jstarks/Qwen3.5-9B-DeepThought-STEM-Weight-Space-Programming run_wsp.py --local-dir ./deepthought-wsp
python ./deepthought-wsp/run_wsp.py --chat # Default Q4_K_M (5.38 GiB, fastest agent)
python ./deepthought-wsp/run_wsp.py --quant q6_k --chat # Q6_K (7.04 GiB, peak 168/270 & 80/150 STEM)
Omit --chat to expose an OpenAI-compatible dynamic routing proxy on http://127.0.0.1:8090/v1, or pass --prompt "..." (--include-bf16 also fetches patches/*.safetensors + HF JSON configs).
llama-server + scripts/deepthought_router.pyllama-server -m Qwen3.5-9B-DeepThought-STEM.Q4_K_M.gguf --host 127.0.0.1 --port 8135 \
-ngl 99 -np 1 -c 16384 -ctk q8_0 -ctv q8_0 -b 512 -ub 256 \
--jinja --reasoning-format deepseek --lora-init-without-apply \
--lora adapters/wsp_s1_think.gguf,adapters/wsp_s2_restraint.gguf,adapters/wsp_s3_router.gguf,adapters/wsp_s4_mid_code.gguf,adapters/wsp_s5_json_schema.gguf,adapters/wsp_s6_handoff.gguf,adapters/wsp_s7_injection_guard.gguf,adapters/wsp_s8_grounding.gguf,adapters/wsp_s9_error_recovery.gguf,adapters/wsp_s10_term_action.gguf,adapters/wsp_s11_term_code.gguf,adapters/wsp_s12_stem_settle.gguf,adapters/wsp_s13_post_concise.gguf
transformers Patching (scripts/apply_patch_12s.py)import torch
from transformers import AutoModelForCausalLM
from scripts.apply_patch_12s import REGIME_PRESETS, apply_subspace_scales_to_hf_model
model = AutoModelForCausalLM.from_pretrained(
"xtools-at/Qwen3.5-9B-DeepThought-STEM",
torch_dtype=torch.bfloat16,
)
apply_subspace_scales_to_hf_model(model, "patches", REGIME_PRESETS["stem_deep_settle"])
Q6_K jumped MATH-500 (13 → 19/50) and MMLU-4C (38 → 41/50) yet scored 3/20 on tbench20: 6-bit precision preserves multi-step BF16 LoRA arithmetic in MATH-500, whereas tbench20 lost heterogeneous-dates on Q6_K and fix-permissions on V4 Q4 (WSP V1 Q4 remains the highest tbench20 run at 5/20).MATH-500 no_box=28 (Q4) failures: Post-</think> S13 cut token length substantially on CAL_MATH_9 (-109t) and MMLU-Pro (641.8t → 601.6t), though on harder Level-5 MATH-500 problems inside stem150, 28 Q4 items still exited or filled 1536 tokens without emitting \boxed{...}.agent100 (86/100 Q4 / 85/100 Q6): V4 brought S6_HANDOFF to 4/4, EST_BFCL_IRR to 12/15, and EST_GSM8K to 17/20, though S4_CODE (1/2 → 0/2 on Q4, 1/2 on Q6) and EST_NL2BASH (18/20 → 17/20) traded 1 task vs V3.xtools-at/Qwen3.5-9B-DeepThought-STEM by xtools-atQ4_K_M & Q6_K): mradermacher/Qwen3.5-9B-DeepThought-STEM-GGUF by mradermacherQwen/Qwen3.5-9Bpatches/wsp_s7_injection_guard_r16.safetensors 5 MBpatches/wsp_s10_term_action_r16.safetensors 5 MBpatches/wsp_s12_stem_settle_r16.safetensors 5 MBpatches/wsp_s13_post_concise_r16.safetensors 5 MBpatches/wsp_s2_restraint_r16.safetensors 5 MBpatches/wsp_s5_json_schema_r16.safetensors 5 MBpatches/wsp_s6_handoff_r16.safetensors 5 MBpatches/wsp_s8_grounding_r16.safetensors 5 MBpatches/wsp_s9_error_recovery_r16.safetensors 5 MBpatches/wsp_s1_think_r16.safetensors 5 MBpatches/wsp_s3_router_r16.safetensors 5 MBpatches/wsp_s11_term_code_r16.safetensors 4 MBpatches/wsp_s4_mid_code_r16.safetensors 4 MB