MoE Sovereign Regulatory Policy & Privacy-by-Design Expert 3B -- SmolLM3 (smollm3-expert-governance-3b)
Model Summary
smollm3-expert-governance-3b is a LoRA fine-tune of the text-decoder of Qwen3.5-4B, specialized as the governance domain expert within the MoE Sovereign compound-AI system.
You are a regulatory policy reasoning and privacy-by-design expert (moe-expert-governance-4b) for GDPR, the EU AI Act, BSI IT-Grundschutz, ISO 27001, and HIPAA. Ground every compliance judgment in the specific article or control it derives from, classify risk levels precisely, and identify privacy-by-design and control-mapping gaps. Never invent a legal citation -- state explicitly when a source is uncertain.
Base Architecture
SmolLM3-3B is a genuinely open-source dense Transformer (weights, training data, and training code all publicly documented by HuggingFaceTB) -- distinguishing this Spur-2 track from the open-weight-only Qwen3.5 base used in the parallel 4B expert line.
Training Configuration
| Parameter | Value |
|---|---|
| Method | LoRA (rank 16, alpha 32, dropout 0.05), targeting q/k/v/o_proj + gate/up/down_proj |
| Trainable parameters | 30,228,480 of 3,105,327,104 (0.97%) |
| Epochs | 3 |
| Effective batch size | 128 (micro-batch 4 x 8 GPUs x grad-accum 4) |
| Learning rate | 1.5e-5 |
| Training sequence length | 4,096 tokens |
| Optimizer sharding | DeepSpeed ZeRO-2, bf16 |
| Compute | EuroHPC LUMI-G, 8x AMD Instinct MI250X GCDs, ROCm |
| Training examples | 2,797 curated instruction/response pairs |
Observed Training Trajectory
Training loss over the run (representative logged steps): 1.707 -> 1.536 -> 1.388. Smooth, monotonic decline consistent with genuine generalization, not memorization.
Prompt Format
ChatML:
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
System Prompt
You are a regulatory policy reasoning and privacy-by-design expert (moe-expert-governance-4b) for GDPR, the EU AI Act, BSI IT-Grundschutz, ISO 27001, and HIPAA. Ground every compliance judgment in the specific article or control it derives from, classify risk levels precisely, and identify privacy-by-design and control-mapping gaps. Never invent a legal citation -- state explicitly when a source is uncertain.
Available Formats
| File | Notes |
|---|---|
smollm3-expert-governance-3b-Q4_K_M.gguf |
Recommended for single/multi-GPU deployment |
smollm3-expert-governance-3b-Q8_0.gguf |
Higher-fidelity reference quantization |
Hardware Guidance
SmolLM3-3B's native context window is 65,536 tokens. On single 8GB GPUs cap num_ctx to 32,768 and use f16 KV-cache on Maxwell-generation hardware (no Flash Attention support there).
Ollama Modelfile
FROM ./smollm3-expert-governance-3b-Q4_K_M.gguf
SYSTEM """You are a regulatory policy reasoning and privacy-by-design expert (moe-expert-governance-4b) for GDPR, the EU AI Act, BSI IT-Grundschutz, ISO 27001, and HIPAA. Ground every compliance judgment in the specific article or control it derives from, classify risk levels precisely, and identify privacy-by-design and control-mapping gaps. Never invent a legal citation -- state explicitly when a source is uncertain."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 32768
Limitations
- Does not execute code/queries/tools itself; outputs should be validated against the actual system before use.
- Specialized for its domain; general-purpose conversation is out of scope.
License
Apache 2.0, inherited from the SmolLM3-3B base model.