AREX-2 — SVD-LoRA (base: Qwen3.8-27B)

The weight difference between BAAI/AREX-2 and stock Qwen/Qwen3.8-27B, compressed into 450 LoRA pairs (rank 64).

AREX-2 is BAAI's 27B long-horizon agent: it proposes a solution, measures it, reflects on the feedback, and revises — over multiple test-time rounds — and that learned self-improvement transfers from coding/ML tasks to deep research.

This is the only adapter of the three companion releases that is Apache-2.0 (Hemmingway-1 and Holo4 carry CC BY-NC), and the only one whose GGUF is complete. Read Known caveats before use — rank 64 captures a smaller share of AREX-2's change than of the others, and that matters for a model whose whole point is multi-round refinement.

---

What AREX-2 is

BAAI's published results for the source model:

None of those numbers were measured on this adapter. They describe the full model whose delta is approximated here.

---

Files

The GGUF is the same adapter converted for llama.cpp — F16 instead of F32, half the size, and this time nothing was dropped: 450 pairs in, 450 pairs out.

---

Measured fidelity

Computed during extraction from the real Δ = W_AREX-2 − W_Qwen3.8-27B; reproducible from extraction_stats.json / validation_report.json.

Scan: 1,184 tensors compared, 450 kept (threshold ‖Δ‖ ≥ 0.05 — AREX-2's fine-tune is very low-amplitude: median ‖Δ‖ across the whole scan is 0.0059, and the largest per-layer delta is 1.34).

Energy retained at rank 64:

This is the most diffuse of the three companion adapters: rank 64 keeps only ~16% of the energy on the MLP and GDN projections, ~25–29% on the attention projections, and the maximum anywhere is 40%. Only the 50 GDN gate pairs (α/β) are exact.

Validation: 40 sampled tensors, adapter-implied delta vs true delta, mean relative error 0.8311.

Honest reading: this adapter reproduces the direction of AREX-2's fine-tune — a broad, low-amplitude adjustment away from base Qwen3.8-27B — but not its details. For a behaviour that lives in long multi-round tool loops (reflect → revise → re-measure), a ~26% energy approximation is a weak instrument. Use it to nudge a base model toward that style, not to replace AREX-2.

---

How it was extracted

Δ = W_AREX-2 − W_Qwen3.8-27B          (base cancels exactly)
Δ ≈ U_r Σ_r V_rᵀ                       (exact truncated SVD, r = 64)
lora_A = Σ_r^½ V_rᵀ ,  lora_B = U_r Σ_r^½

1. Architecture check — both sides: qwen3_5, 64 layers, hidden 5120, intermediate 17408, vocab 248320, GDN linear attention on 48 of 64 blocks. Key names already match (model.language_model.layers.N.*) — no remapping needed.
2. Norm scan — 1,184 tensors. Largest delta in the model: lm_head.weight at ‖Δ‖ = 2.88 (about 2× the largest layer delta); embed_tokens 1.09. Both skipped — see caveat 3.
3. Exact truncated SVD — target rank 64 per tensor, capped at each matrix's own rank (48 for the α/β gates) → 450 pairs.
4. Validation — 40-tensor sample.
5. Export — float32 PEFT, and an F16 llama.cpp GGUF with blk.N.* names (ssm_alpha, ssm_beta, ssm_out, attn_qkv, attn_gate, ffn_*, attn_*).

No AREX-2 training data, trajectories or reward models are involved — this is arithmetic on two open checkpoints.

---

Usage

llama.cpp / llama-server (GGUF)

llama-server \
  -m Qwen3.8-27B-Q6_K.gguf \
  --lora AREX-2_qwen3.8-27B_svd_lora.gguf \
  -ngl 99 -c 65536 -fa on --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080

Pair with any Qwen3.8-27B GGUF, e.g. unsloth/Qwen3.8-27B-GGUF. The graph of this architecture applies LoRA to the GDN gates (ssm_alpha, ssm_beta, ssm_out, wqkv, wqkv_gate) — tensors that are exact in this adapter.

Partial strength:

--lora-scaled AREX-2_qwen3.8-27B_svd_lora.gguf:0.5

Long context is the point of this model: give it room (-c 65536+, quantised KV if VRAM is tight).

Python — PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.8-27B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "hotdogs/AREX-2_qwen3.8-27B_svd_lora")

With the shipped config, loading raises a size mismatch on the gate tensors — apply the rank_pattern fix in caveat 1 first.

Merging

merged = model.merge_and_unload()      # after the rank fix
merged.save_pretrained("AREX-2-lora-merged")

Apache-2.0 throughout, so a merged model can be redistributed under the same terms (keep the upstream notices for the Qwen base).

Agent loop

AREX-2's intended shape is: propose → run → read scores/logs/errors → revise, over several rounds. With llama.cpp you provide the tool schema and the loop:

curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "arex2-lora",
  "messages": [
    {"role":"system","content":"You improve a solution over multiple rounds. After each attempt you will receive scores and errors; reflect, then revise."},
    {"role":"user","content":"Write a Python function that beats the baseline on the provided metric. Round 1: propose; you will get feedback before round 2."}
  ],
  "max_tokens": 4096,
  "chat_template_kwargs": {"reasoning_effort": "medium"}
}'

Feed the real metrics back in the next turn — that is where this model's behaviour lives, and where a 26%-energy approximation will show its limits.

---

Known caveats

Caveat 1 — mixed rank vs. one global rank/alpha

The SVD produced rank 64 for most pairs and rank 48 for in_proj_a / in_proj_b (their matrices are 48 × 5120, so 48 is full rank — hence their exact reconstruction). The config declares a single r = 64, lora_alpha = 64.

The correct scale is 1.000 for every pair — verified directly: for a rank-48 pair, ‖B·A‖ / ‖Δ‖ = 1.0000, i.e. the stored product already is the delta.

• llama.cpp computes scale = alpha / rank per tensor (llama_adapter_lora_weight::get_scale(), rank inferred from lora_b), so those 50 pairs run 33% over-strength.
• PEFT builds layers from the config (nn.Linear(in_features, r)), expects lora_A of shape (64, 5120), finds (48, 5120) → default PeftModel.from_pretrained errors out; with ignore_mismatched_sizes=True they are silently dropped (gates do nothing).

Fix (PEFT):

from peft import LoraConfig

cfg = LoraConfig(
    r=64, lora_alpha=64,
    rank_pattern={"in_proj_a": 48, "in_proj_b": 48},
    alpha_pattern={"in_proj_a": 48, "in_proj_b": 48},
    target_modules=["down_proj","gate_proj","in_proj_a","in_proj_b","in_proj_qkv",
                    "in_proj_z","k_proj","o_proj","out_proj","q_proj","up_proj","v_proj"],
)

Fix (llama.cpp) — rescale lora_B on the affected pairs so 4/3 × B'·A = B·A, then re-convert to GGUF:

from safetensors.torch import load_file, save_file

sd = load_file("adapter_model.safetensors")
for k in list(sd):
    if k.endswith(("in_proj_a.weight.lora_B.default", "in_proj_b.weight.lora_B.default")):
        sd[k] = sd[k] * 0.75          # 48 / 64
save_file(sd, "adapter_model_fixed.safetensors")

A global --lora-scaled cannot fix this — it would rescale the rank-64 pairs too.

Caveat 2 — the GGUF is complete (the good news)

PEFT 900 tensors = 450 pairs; GGUF 900 tensors = 450 pairs. Names map 1:1 (in_proj_a→ssm_alpha, in_proj_b→ssm_beta, in_proj_qkv→attn_qkv, in_proj_z→attn_gate, out_proj→ssm_out, self_attn.o_proj→attn_output, mlp.*→ffn_*). No dropped pairs — unlike the Holo4 companion release, whose GGUF loses its two conv1d pairs.

Caveat 3 — embeddings and head are not in the adapter

By design (skipped_large): a full SVD of a 248,320 × 5,120 matrix did not fit the memory the extraction ran in. Here the omission is milder than for Holo4 — the head delta is ~2× the biggest layer delta rather than ~45× — but any behaviour that lives in the embedding table or the output head is not represented. Vocabularies match exactly (248,320 tokens both sides), so this is a weight change, not a tokenizer change.

Caveat 4 — vision

AREX-2 is a multimodal checkpoint, and the language tower is what this adapter covers (exclude_visual_tower). Image understanding comes from whichever Qwen3.8-27B vision stack you pair it with, not from AREX-2's training.

Caveat 5 — expectations

Rank 64 keeps 26.5% of the delta energy on average and never more than 40% outside the gates. Expect a partial shift toward terse, iterative, feedback-driven reasoning on a base Qwen model — not AREX-2's benchmark scores, and not its reliability in long agent loops. No benchmark has been run on this adapter.

---

License

Apache-2.0 — this adapter inherits AREX-2's licence, which is Apache-2.0 (the most permissive of the three companion releases). The Qwen3.8-27B base is Apache-2.0 as well, so merged derivatives can be redistributed freely, keeping the upstream notices and the paper citation.

You are still responsible for how the merged model behaves.

---

Credits

• Source model: BAAI/AREX-2 — VectorSpaceLab, Beijing Academy of Artificial Intelligence. All the training, data and benchmarks are theirs.
• Base model: Qwen/Qwen3.8-27B (Apache-2.0)
• Paper: Qian et al., AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks, arXiv:2609.38288
• Method: weight-diff exact truncated SVD per tensor on identical architectures.
• Runtime: llama.cpp / PEFT

Not affiliated with or endorsed by BAAI or Alibaba Cloud.

Citation

If you use this adapter, cite the model it came from:

@article{2026arex2,
  title   = {AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks},
  author  = {Qian, Hongjin and Li, Chaofan and Luo, Kun and Wei, Wenqing and Chen, Jianlyu and Lu, Shuqi
             and Hu, Yuyang and Xiao, Hongwang and Wang, Hui and Li, Chaozhuo and Ye, Qiwei
             and Dou, Zhicheng and Lian, Defu and Liu, Zheng},
  journal = {arXiv preprint arXiv:2609.38288},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.38288}
}

@misc{hotdogs_arex2_svd_lora,
  title  = {AREX-2 — SVD-LoRA (base: Qwen3.8-27B)},
  author = {hotdogs},
  year   = {2026},
  note   = {Weight-diff exact-SVD adaptation of BAAI/AREX-2 onto Qwen/Qwen3.8-27B,
            rank 64, 450 LoRA pairs},
  url    = {https://huggingface.co/hotdogs/AREX-2_qwen3.8-27B_svd_lora}
}

---

ภาษาไทย

นี่คืออะไร

LoRA ของ AREX-2 — โมเดล agent 27B ของ BAAI (VectorSpaceLab) ที่เก่งงาน "แก้ปัญหาซ้ำหลายรอบ": เสนอวิธี → วัดผล → อ่าน feedback (คะแนน/ล็อก/error) → ปรับปรุง แล้ววนซ้ำ และความสามารถนี้ถ่ายทอดไปงาน deep research ได้

adapter นี้คือส่วนต่างน้ำหนักระหว่าง AREX-2 กับ Qwen3.8-27B บีบด้วย SVD rank 64 → 450 คู่

จุดเด่นของตัวนี้: เป็นตัวเดียวในสามตัวที่ Apache-2.0 (Hemmingway / Holo4 เป็น CC BY-NC) และ GGUF ครบทุกคู่ (900 เทนเซอร์ = 450 คู่ ไม่ขาดอะไร)

ไฟล์

วิธีใช้

llama-server -m Qwen3.8-27B-Q6_K.gguf \
  --lora AREX-2_qwen3.8-27B_svd_lora.gguf \
  -ngl 99 -c 65536 -fa on --jinja --cache-type-k q8_0 --cache-type-v q8_0

from peft import PeftModel
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B",
                                             dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "hotdogs/AREX-2_qwen3.8-27B_svd_lora")

⚠️ PEFT โหลดด้วย config เดิมจะ error — ต้องใส่ rank_pattern/alpha_pattern = {"in_proj_a": 48, "in_proj_b": 48} ก่อน (ดู Caveat 1)

ความแม่นยำที่วัดได้จริง (ต้องอ่านก่อนใช้)

• พลังงานที่เก็บได้เฉลี่ย 26.5% — น้อยที่สุดในสามตัว
  • MLP / projection: ~14–17%
  • attention (q/k/v/o): ~17–29% (สูงสุดในไฟล์แค่ 40%)
  • ประตู GDN (α/β, 50 คู่): 100% (ตรงเป๊ะ)
• ค่าเฉลี่ย relative error ที่ตรวจ 40 เทนเซอร์ = 0.831
• lm_head (‖Δ‖ = 2.88) และ embed_tokens (1.09) ถูกตัดออก → พฤติกรรมที่อยู่ใน embedding/head ไม่มีใน adapter นี้
• สรุปตรง ๆ: นี่คือ "ทิศทาง" ของ AREX-2 ไม่ใช่ตัวโมเดล — ใช้เพื่อดึง base Qwen ให้คิดแบบวนซ้ำ/อ่าน feedback ได้ แต่ อย่าหวังคะแนน benchmark ของ AREX-2 (ยังไม่มีใครรัน benchmark กับ LoRA ตัวนี้)

ข้อควรระวัง

1. rank ไม่ตรง config: 50 คู่เป็น rank 48 แต่ config บอก r=64
   • llama.cpp: คู่ rank 48 ถูกใช้เกิน 33% → คูณ lora_B คู่นั้นด้วย 0.75 แล้วแปลง GGUF ใหม่
   • PEFT: ต้องใส่ rank_pattern/alpha_pattern ไม่งั้นโหลดไม่ผ่าน
2. ไม่รวม vision tower (AREX-2 เป็น multimodal) → ใช้ vision ของ Qwen base แทน
3. ใบอนุญาต Apache-2.0 → ใช้เชิงพาณิชย์ได้ (แต่ให้เครดิต BAAI + อ้าง paper)