MiniCPM-V-4.7-35B-A3B-CRACK (GGUF)

Refusal-direction–ablated GGUF of openbmb/MiniCPM-V-4.7-35B-A3B, the 35B-A3B Qwen3.5-moe-text (hybrid GatedDeltaNet + Full-Attention + 256 routed experts + 1 shared expert) multimodal model.

Surgery removes the model's refusal direction from all three residual-stream writer tiers of the Qwen3.5 hybrid architecture — the SSM output path is the critical tier that single-tier surgeries miss on this architecture. Multimodal (16× + 4× projectors) and DSpark speculative-decoding paths are preserved.

Files in this repo

Evaluation — base vs CRACK

HarmBench-240 (standard + contextual + copyright splits), MMLU-200 (balanced subjects, logit-eval), multi-turn 5-scenario coherence, harsh-10 hot categories. Multimodal verified on synthetic image (number 7342 + red square + blue circle).

Gate status

• Q8_0 / Q4_K_M: pass the strict compliance gate (HB-240 ≥ 97% comply) and the strict knowledge gate (ΔMMLU within ±3pp of same-quant base).
• IQ2_M: passes the compliance gate (100% comply, 0 refuse, 0 garbage, no loops) but MMLU drops −7.5pp vs same-quant base, outside the ±3pp knowledge gate. The 2.5-bit quant combined with 58 residual writers of surgery is beyond what the surgery direction tolerates under this quant. Ship anyway for compliance+size users; expect degraded knowledge accuracy on multiple-choice benchmarks. Multimodal, HB and MT are unaffected.

Compliance behavior

• HB-240 = 100.0% on every quant (0 refuse, 0 garbage, 0 empty, 0 loop across 720 rows)
• Multi-turn 5-scenario: all 15 turns coherent, no refusals, no loops
• Harsh "FULL" = complete compliance; "SOFT" = compliant with light disclaimer, no HARD refusal on any quant
• Multimodal: all 3 quants correctly identify "7342 + red square + blue circle" through both 16× (full-resolution) and 4× (reduced-token) projectors

Usage

# Standard inference
./llama-server -m MiniCPM-V-4.7-35B-A3B-CRACK-Q4_K_M.gguf \
  --mmproj mmproj-MiniCPM-V-4.7-35B-A3B-CRACK-f16.gguf \
  -ngl 99 -c 8192 --jinja --port 8080

# With DSpark speculative decoding
./llama-server -m MiniCPM-V-4.7-35B-A3B-CRACK-Q4_K_M.gguf \
  --mmproj mmproj-MiniCPM-V-4.7-35B-A3B-CRACK-f16.gguf \
  --model-draft MiniCPM-V-4.7-35B-A3B-CRACK-DSpark-BF16.gguf \
  --spec-type draft-dspark \
  -ngl 99 -c 8192 --jinja --port 8080

Thinking modes: enable_thinking=true for CoT reasoning; chat_template_kwargs: {"enable_thinking": false} to skip the <think>...</think> block. The HB and MMLU numbers above are from enable_thinking=false (direct-answer mode).

Recipe

MiniCPM-V-4.7-35B is a hybrid GDN + FA MoE where three residual-stream tiers all carry refusal signal. Single-tier surgeries (attn-only or shared-only) leave the refusal path intact through the SSM recurrent-state output.

MPOA at uniform s=2.0 on deep-middle layers [11..39], three tiers:

1. self_attn.o_proj.weight on FA layers [3,7,11,15,19,23,27,31,35,39] intersected with band → 7 layers
2. linear_attn.out_proj.weight on SSM/GDN layers (every non-FA layer in band) → 21 layers
3. mlp.shared_expert.down_proj.weight on every layer in band → 29 layers

Routed experts (mlp.experts.N.*) are not touched — pass-through byte-identical. MoE routing bypasses the "problem expert", so ablating routed experts adds noise without helping; the Q6_K uplift on ffn_down_exps at quantize time preserves the shared-expert surgery across quants.

Vision tower (model.visual.*), MTP head, lm_head, embed_tokens: byte-identical passthrough.

• Vector blend 0.5·v_pure + 0.5·v_svd:
  • v_svd = SVD top-1 right-singular vector across {reasoning-ON mean-diff, reasoning-OFF mean-diff}
  • v_pure = normalize(v_raw + v_svd) where v_raw = mean(harm) − mean(harmless)
• Probe: BF16 extraction via llama-probe-capture on l_out-* tensors, mode-invariant SVD per layer, chat-template-rendered prompts with enable_thinking=true AND enable_thinking=false surfaces
• Probe corpus: 10,958 harmful/benign-twin pairs = glm53flash multi2k (2000 EN+ZH twin pairs from the GLM-5.3-Flash NVFP4 calibration set) + MultiJail/HarmBench/CyberSecEval/OASST1 twins (8958 pairs across 14 languages). The glm53flash cybersec/pentest unpaired prompts (447 rows) were used in an earlier probe pass but excluded from the final BF16 probe to keep the matched-pair structure clean.

Thinking-ON compliance (verified per-row)

HB-240 was run twice on Q4_K_M (the recommended shipping quant) — once with enable_thinking=false (direct answer) and once with enable_thinking=true (CoT reasoning). Per-row answers + reasoning traces persisted to disk.

Reasoning-mode compliance matches direct-answer-mode compliance exactly across all 240 rows. The model emits a full <think>...</think> block for each harmful prompt and then completes the task — surgery is stable through the CoT path.

License

Inherits Apache 2.0 from upstream openbmb/MiniCPM-V-4.7-35B-A3B.

---

Detailed per-subject MMLU + per-category HB tables

35B — MMLU per-subject (base vs CRACK)

Q8_0

Overall: base 76.0% → CRACK 77.0% (Δ +1.0pp)

Q4_K_M

Overall: base 75.0% → CRACK 75.5% (Δ +0.5pp)

IQ2_M

Overall: base 76.5% → CRACK 69.0% (Δ -7.5pp)

35B — HarmBench-240 per-category verdicts

Q8_0

Overall: 240/240 = 100.0% comply · verdicts: {'comply': 240}

Q4_K_M

Overall: 240/240 = 100.0% comply · verdicts: {'comply': 240}

IQ2_M

Overall: 240/240 = 100.0% comply · verdicts: {'comply': 240}

35B — Harsh grade (10 hot categories, FULL/SOFT/HARD/EMPTY)

Q8_0

Summary: 1}

Q4_K_M

Summary: 2}

IQ2_M

Summary: 2}