agent-release-gate · Qwen3-4B-Instruct-2507 LoRA (MLX)

A small LoRA adapter that turns Qwen3-4B-Instruct-2507 into a release gate for changes to AI agents and automations: given a change request, it returns JSON with a risk class, the required gates and a short rationale. Trained with mlx-lm on Apple Silicon, on the synthetic NovaeonStudio/agent-release-gate dataset.

Results (held-out test split, n=341, greedy decoding)

Model	Risk accuracy	Gate micro-F1	Gate exact match	Valid JSON
Qwen3-4B-Instruct-2507 (4-bit), zero-shot, short prompt	0.513	0.728	0.194	1.000
Qwen3-4B-Instruct-2507 (4-bit), zero-shot, full rubric in prompt	0.545	0.766	0.320	0.994
+ agent-release-gate LoRA (short prompt)	0.633	0.866	0.478	1.000

"Short prompt" = the one-line system prompt below, used in training. The rubric row shows what the base model reaches when it is given the full labelling rubric in the prompt.

Usage

from huggingface_hub import snapshot_download
from mlx_lm import load, generate
adapter = snapshot_download("NovaeonStudio/agent-release-gate-qwen3-4b-lora")
model, tok = load("mlx-community/Qwen3-4B-Instruct-2507-4bit", adapter_path=adapter)
system = "You are a release gate for changes to AI agents and automations. Classify the change request. Answer with JSON only: {\"risk\": \"low|medium|high|critical\", \"gates\": [...], \"rationale\": \"...\"}. Allowed gates: regression_tests, security_review, peer_review, business_approval, cab_approval, rollback_plan, staged_rollout."
msgs = [{"role": "system", "content": system},
        {"role": "user", "content": "Swap the invoice-matching agent in production from model A to model B ..."}]
print(generate(model, tok, tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False), max_tokens=200))

The adapter targets the base model's 4-bit MLX build; it was not fused or tested against other quantisations.

Training

LoRA via mlx_lm lora, base mlx-community/Qwen3-4B-Instruct-2507-4bit, 16 layers, batch 4, lr 1e-4, max length 1024, prompt masked, LoRA rank 8, scale 20, 1,200 iterations (~1.6 epochs over 2,926 training rows, with 153 train-split rows held out for validation; these counts are from before the 8-row privacy screen; final val loss 0.856, best 0.833 at iteration 500), about 20 minutes on an Apple M5 Max, peak system memory 51 GB.

Limitations

Fully synthetic. Written by a language model from a fixed rubric; no real change records, customers, systems or people. Phrasing is cleaner and more uniform than real tickets.
Labels encode one rubric, not an industry standard. Real organisations weigh risk differently (regulation, CAB culture, tooling). Treat it as a starting point, not as compliance advice.
English only. SAP modules and a few well-known product names appear only as generic context; all organisations, teams, systems and ids are invented.
Output may start with an empty <think></think> block from the chat template; parse the first {...} JSON object. The labels are usually the strongest part; rationales are short and can be internally inconsistent (e.g. we saw one that called a change critical and then said it was "not critical").
A decision aid, not an approver: route anything high/critical to humans.

License: Apache-2.0 (same as the base model). Built by Novaeon Studio, 2026.