Qwen3.5-2B-ita-agent-MLX-8bit

An Italian agent for MCP tools and the terminal: Qwen3.5-2B fine-tuned with LoRA on verified synthetic conversations, fused and quantized to 8-bit MLX. It reads and writes files, manages notes and calendar events, computes dates and VAT with tools, fetches web pages, asks before deleting or changing data, and answers plain chat in Italian. It is not a coding model.

Everything was done at home on two Mac mini M4 with 16 GB: data generation, training and evaluation. No rented GPU, no closed model output in the training data.

Results

All numbers are greedy decoding, measured on this fused 8-bit model unless stated otherwise. Tasks run in deterministic simulated environments served as real MCP servers (files, notes, calendar, web, terminal, Italian date and tax tools); a task passes only if the environment state and the answer pass their checks.

The dev split has 10 categories of 8 to 10 tasks: single call, chain, multi-turn, parallel calls, no tool needed, clarify, recovery after a tool error, summarize, choice, safety. Fusion quantizes the weights again: the adapter before fusion scored 66/88 on dev and 230/296 on fresh5, and the fused model differs from it on 1 dev and 5 fresh5 tasks, within noise. Safety (ask before deleting or overwriting): 10/10 on dev, 24/26 on fresh5.

The frozen test split is the main number: 112 tasks written before training and read once, after every choice was made, 45 of them on two environments the model never saw in training (TLS certificates and git). Dev guided the harness and the training data, so it overstates this model by about 15 points. On the test, stock Qwen3.5-4B 8-bit with the same harness passes 72/112: this model is below it, though not significantly (p = 0.43).

The harness relies on turn labels this model was trained to produce, and the stock 2B does better without it (58/112) than inside it (48/112). Against the stock 2B run plainly, as most people run it, this model with its harness gains little on tasks: 66 against 58, 23 tasks won and 15 lost, p = 0.26, within noise. Its gain is in safety: it asks before a destructive action in 9 of 11 test safety tasks, against 4 of 11 for the plain stock 2B, 3 of 11 for the stock 2B in the harness and 3 of 11 for the stock 4B.

The agent harness

The best numbers come with a thin harness around the model, part of the training setup:

• each user turn is first classified by the model itself as answer, tools, ask or confirm, and the turn then runs with that intent;
• in an ask or confirm turn, tools that delete or change data are hidden and refused;
• file editing uses replace_in_file and append_to_file instead of whole-file rewrites, and a JSON or Python file read or written gets a syntax verdict;
• the context lists the calendar from today for the next seven days.

Without the harness the model still calls tools in the trained format, with the lower score in the table. Safety is where the harness matters most: without it the model asked before a destructive action in 5 of 10 dev safety tasks, against 10 of 10 with it.

Training

• Base: Qwen/Qwen3.5-2B, 8-bit MLX (group size 64), the model's own chat template and the qwen3_coder tool-call format.
• LoRA rank 16, scale 20, all layers, learning rate 2e-5 with 30 warmup steps, one epoch, sequences up to 3,072 tokens; 16.8M trainable parameters; 3,322 steps in about 8.4 hours on two Mac mini M4 16 GB as a ring (gradients averaged every step), with MLX.
• Data: 2,567 conversations and 3,576 turn-classification examples, generated by Qwen3.5-4B (8-bit, Apache-2.0) acting in 61 task families of simulated Italian worlds, with tool faults injected in part of the tasks. A conversation is kept only if the environment checks pass. Evaluation tasks and the test-only environments never appear in the data; a contamination check of the test requests against the data found no copies.
• Final validation loss 0.103.

Use

from mlx_lm import load, generate

model, tokenizer = load("fabriziosalmi/Qwen3.5-2B-ita-agent-MLX-8bit")
tools = [{
    "type": "function",
    "function": {
        "name": "list_directory",
        "description": "Elenca file e cartelle in un percorso",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string", "description": "Percorso della cartella"}},
            "required": ["path"],
        },
    },
}]
messages = [{"role": "user", "content": "Che file ci sono in /home/utente/documenti?"}]
prompt = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, enable_thinking=False
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

tools holds the JSON schemas of the MCP tools you connect. A tool call comes back in the qwen3_coder format of the base model; your client runs it and sends the result back as a tool message. Keep enable_thinking=False: the model was trained and evaluated without thinking.

Limitations

• Measured on simulated environments, not yet on real MCP servers.
• Weakest areas on the test: chains of dependent calls (2 of 10), clarifying an ambiguous request (3 of 10) and recovery after a tool error (4 of 10).
• Without the harness it asks before a destructive action only half of the time: keep destructive tools behind a confirmation in your client.
• Italian only. Thinking mode was off in training and evaluation.

License

Apache-2.0, as the base model and the teacher.