Nexus Swarm Orchestrator — DPO Adapter (Qwen3-14B)

LoRA adapter trained with Direct Preference Optimization (DPO) on Qwen/Qwen3-14B for the orchestrator role in the NEXUS Swarm.

This is the DPO stage — the final RL step in the orchestrator training pipeline:

DAPT → SFT → FARCA-GRPO → DPO → GGUF

DPO Method

DPO aligns the model with human-preferred response patterns using contrastive preference pairs. The orchestrator DPO dataset includes:

• Offensive security preference pairs — correct tool selection and structured reasoning vs. suboptimal approaches
• Nexus persona alignment pairs — maintaining the Nexus identity and ASSESS → SELECT → ACT reasoning protocol
• Conciseness pairs — tight reasoning over verbose explanation

The reference model uses PEFT implicit mode (LoRA disabled for reference forward passes) since two copies of the 14B model would exceed A100 80 GB VRAM.

Training Details

Post-RL Validation Gate

• Tool-call pass rate ≥ 0.80 across 14 scenarios
• Agent progress ≥ 5 across 7 agentic scenarios

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B", torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "atlas-institute/qwen3-14b-orchestrator-dpo")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

messages = [
    {"role": "system", "content": "You are Nexus, an offensive security orchestrator agent."},
    {"role": "user", "content": "Analyze the nmap results and prioritize exploitation targets."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=768)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Related Models

Developed by

Atlas Institute · Part of the Code-Trainer (RTPI) project.