MoE Sovereign Planner 7B -- OLMo-3 (moe-sovereign-planner-olmo3-7b)
Task Decomposition & Orchestration
Model Summary
moe-sovereign-planner-olmo3-7b is a LoRA fine-tune of OLMo-3-7B-Instruct, specialized as the orchestrator/planner of the MoE Sovereign compound-AI system: it decomposes an incoming request into 1-4 subtasks for the domain experts, extracting and propagating explicit numerical constraints so experts cannot hallucinate default values.
Genuinely open-source base (Ai2 -- weights, training data, and training code all publicly documented), the Spur-2 (open-source) counterpart to the parallel Qwen3.5-9B planner trained on the open-weight track.
Training Configuration
| Parameter | Value |
|---|---|
| Method | LoRA (rank 16, alpha 32, dropout 0.05) |
| Trainable parameters | 39,976,960 of 7,337,988,096 (0.54%) |
| Epochs | 3 |
| Effective batch size | 128 (micro-batch 4 x 8 GPUs x grad-accum 4) |
| Learning rate | 1.5e-5 |
| Training sequence length | 4,096 tokens |
| Compute | EuroHPC LUMI-G, 8x AMD Instinct MI250X GCDs, ROCm |
| Training examples | 4,726 curated decomposition examples |
Observed Training Trajectory
Training loss: 2.111 -> 1.094 -> 0.737. Smooth decline, no overfitting signature.
Prompt Format
ChatML. System prompt:
You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.
Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values.
Available Formats
| File | Notes |
|---|---|
moe-sovereign-planner-olmo3-7b-Q4_K_M.gguf |
Recommended for deployment |
moe-sovereign-planner-olmo3-7b-Q8_0.gguf |
Higher-fidelity reference quantization |
Hardware Guidance
OLMo-3-7B's native context is 65,536 tokens (via YaRN extension). Fits comfortably on a single modern 12GB+ GPU at Q4_K_M.
Ollama Modelfile
FROM ./moe-sovereign-planner-olmo3-7b-Q4_K_M.gguf
SYSTEM """You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.
Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 32768
Limitations
- Decomposition quality depends on the request containing extractable constraints; ambiguous requests may yield underspecified subtasks.
- Does not execute the subtasks itself -- routes to the domain experts.
License
Apache 2.0, inherited from the OLMo-3-7B-Instruct base model.