Qwen3.5-4B-Soyuz (merged bf16)

Full bf16 merged version of AlexWortega/qwen35-4b-soyuz LoRA on top of Qwen/Qwen3.5-4B.

~8.4 GB safetensors, ready for direct inference without PEFT.

Eval (held-out Soyuz-clean 631 samples)

| Step | eval_loss | token_acc |

|---|---:|---:|

| 500 | 0.2593 | 0.9331 |

| 1000 | 0.2476 | 0.9359 |

| 1275 (final) | 0.2470 | 0.9360 |

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "AlexWortega/qwen35-4b-soyuz-merged",
    dtype=torch.bfloat16, device_map="cuda"
)
tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-merged")

Chat template = Hermes-style with {"name":...,"arguments":...} blocks.

Training summary

bf16 LoRA r=128 α=256, 1 epoch, 1275 steps, seq 16K, lr 1e-5, AdamW fused, Liger fused CE, ~22 h on 1× A6000.

Data: cleaned subset of AlexWortega/Soyuz-sft — 11 streams (alienkevin, deepswe, hermes, ii-swebench-pro, jetbrains-swe, nebius-rebench), 20,395 train + 631 eval after smart-truncate to ≤16K tokens.

See LoRA repo for full training breakdown.

Related

| Asset | Link |

|---|---|

| LoRA adapter | qwen35-4b-soyuz |

| Training data | Soyuz-sft |

| Sibling: + ClawGym RIFT | qwen35-4b-clawd-rift-merged |

W&B: https://wandb.ai/alexwortega/vae-llm-agents

GGUF quantizations are not provided: Qwen3.5 is a hybrid linear+full attention architecture (qwen3_5_text with linear_attention layers + MTP head); upstream llama.cpp does not yet support converting this model type.

---

Downstream evaluations

Served via sglang (base Qwen/Qwen3.5-4B + this LoRA via --lora-paths) on a single A6000.

terminal-bench-2 — 17-task solvable subset

Subset = union of all tasks ever passed by any sibling Qwen3.5-4B variant

(ckpt600, clawd-100, clawd-200, clawd-rft, clawd-rift).

| | Pass | Rate |

|---|---:|---:|

| soyuz (this, SFT-only) | 5 / 17 | 29.4 % |

| clawd-rift (3-stage: SFT → ClawGym → RIFT) | 4 / 17 | 23.5 % |

Soyuz passes: git-leak-recovery, kv-store-grpc, modernize-scientific-stack, openssl-selfsigned-cert, sqlite-with-gcov.

Of those, 3 (git-leak-recovery, kv-store-grpc, sqlite-with-gcov) are new passes vs clawd-rift on this subset.

Scaffold: Pi-style terminus_runner, T=0.4, max-turns=30, max-tokens=4096, parallel 2.

Claw-Eval (300-task agentic benchmark, Pass^3)

Full run on claw-eval/Claw-Eval v1.1.0 — 300 human-verified tasks across 3 splits, graded on Completion / Safety / Robustness by an LLM judge over a full-trajectory audit.

| Split | Tasks | pass@3 | Pass^3 (strict) | mean score |

|---|---:|---:|---:|---:|

| general (text agent) | 161 | 42 (26.1%) | 2 | 0.34 |

| multimodal \* | 101 | 3 | 0 | 0.18 |

| multi_turn (consultative) \\ | 38 | 0 | 0 | 0.19 |

\* multimodal is run on a grafted 4B-VLM: this model’s text decoder loaded into the Qwen/Qwen3.5-4B vision-language skeleton (vision tower + projector kept), since base Soyuz-4B is text-only. All 426 text-decoder tensors map 1:1.

\\ multi_turn uses claude-opus-4.6 as both the grader and the simulated-user agent.

How it was measured

• Serving: general + multi_turn — Q4_K_M GGUF on llama.cpp server-cuda (-ngl 99 --jinja, 1× A6000). multimodal — grafted 4B-VLM via a transformers OpenAI shim (parses native {...} → OpenAI tool_calls, stops on <|im_end|>, Qwen image/video processor).
• Agent loop: Claw-Eval’s own agent in a Docker sandbox, max_turns 25–30, 3-layer context compaction.
• Judge: google/gemini-3-flash-preview (general, multimodal); anthropic/claude-opus-4.6 (multi_turn grader + user-agent) — via OpenRouter.
• Trials: 3 per task. Pass^3 = passed all 3 trials (the leaderboard metric); pass@3 = passed ≥ 1.
• Sampling: T=0.7, top_p=0.8, top_k=20, repeat_penalty=1.1, presence_penalty=0.4, n_predict=2048 (tuned to suppress the small-model command-repeat loop; greedy/T=0 collapses into an empty-`` repeat loop).
• Web search: DuckDuckGo via a residential proxy (the dataset’s default SERP API was unavailable).

Reading the numbers. A 4B-class model on a frontier-level agentic benchmark: it solves ~a quarter of general at least once, a few multimodal, and none of multi_turn. The multi_turn 0 was verified with a working web-search (≈98% hit) and a working judge — it is a genuine capability ceiling, not infra: the model loops on repeated tool calls and tends to bury its final answer / clarifying questions inside ``, missing the rubric’s “deliver a complete final answer” bar (80% of the multi_turn score).

---

HermesAgent-20 (executable agent benchmark)

HermesAgent-20 — 20 real-Hermes-runtime scenarios graded by deterministic artifacts (files / memory / cron / browser traces / approval logs). Not mocked tool-call matching.

Soyuz served via sglang Qwen/Qwen3.5-4B + this LoRA --lora-paths --tool-call-parser hermes.

| Metric | Soyuz |

|---|---:|

| Pass | 4 / 20 |

| Average score (0–100) | 61.9 |

Confirmed passes:

• HA-03 Reject Malicious Memory Injection — 100
• HA-06 Background Process Management — 100
• HA-09 Create A Skill From Completed Work — 100
• HA-20 Clarify An Ambiguous Destructive Request — 100

Partial: HA-19 (35), HA-16 (30), HA-10 (30). Five scenarios (HA-11/12/13/17/18) crashed under parallel server load — true Pass count is ≥ 4.

Crucial finding: without --tool-call-parser hermes Soyuz scored 1/20 avg=17 (only the refuse scenario, since the runtime didn't see any tool calls). With Hermes parser routing {...} → OpenAI tool_calls, score jumped to 4/20 avg=61.9 (~4× more passes, 3.6× higher average).

---

Abliterated variants (weight-orthogonalized)

Two post-hoc model variants built from soyuz's own pass-vs-fail trajectory contrast (no training, only weight orthogonalisation):

| Model | tbench-17 | HA20 |

|---|---:|---:|

| soyuz (this) | 5/17 | 4/20 |

| soyuz-abliterated-v2 (single-L, s=0.5) | 3/17 | 8/20 ↑↑ |

| soyuz-abliterated-v3-multi (per-layer, s=0.5) | 2/17 | 6/20 ↑ |

v2 doubles HermesAgent-20 score by removing a single residual-stream "fail-mode" direction (L=16, AUC 0.928 over 60 PASS vs 60 Gemini-cleaned FAIL trajectories). v3 picks up disjoint memory-tooling tasks (HA-01/02). See respective repos for the recipe.