Qwen3-0.6B-tools GGUF

The Qwen3-0.6B tool-calling fine-tune in GGUF format, for llama.cpp-style runtimes. The model was supervised-fine-tuned with LoRA so it reliably emits OpenAI-style tool calls for agentic workflows. The full BF16 model and its validation table live at iromu/Qwen3-0.6B-tools.

Base model

This model is the GGUF conversion of:

iromu/Qwen3-0.6B-tools

— itself a LoRA fine-tune of Qwen/Qwen3-0.6B.

GGUF files

The model is provided in GGUF format at the following precisions:

The uniform quants above are joined by an imatrix-guided ("Dynamic") family, UD-Q2_K_XL through UD-Q8_K_XL. These allocate bits per tensor by activation importance rather than uniformly, so a file lands near the size of its nominal tier with better fidelity:

The importance matrix behind the UD-* family was calibrated on the SFT train split under this model's chat template — never on the validation split the validation table below scores on.

On this model the exact-args fidelity (measured on the sft_tools validation split; full numbers in the validation table below) is not a smooth ramp in quant size. Q4_K_M drops to about a third of the BF16 rate; Q5_K_M and Q6_K sit in the same ~54% band, so Q6_K costs more bytes for no fidelity gain over Q5_K_M; and Q8_0 is the fidelity pick — it reaches parity with the BF16 conversion (~66% exact args) and is the quant to use when the exact tool arguments matter.

Training

Supervised fine-tuning with LoRA, run with NeMo-RL, assistant-turn-masked cross-entropy over the tool-calling logs.

• LoRA: dim 32, alpha 32, dropout 0.05, target modules *.proj (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• Max sequence length: 4096
• Precision: bfloat16
• Optimizer: AdamW, lr 5e-5, weight decay 0.01
• Schedule: 15-step linear warmup, then cosine decay to 1e-6
• Global batch size: 64 (micro batch 2)
• Training steps: 336 (validation every 84)
• Loss masked to assistant turns via the chat template's generation keyword

Dataset

Training used the sft_tools split of the r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset (OpenAI messages format; assistant turns carry tool_calls).

Intended use

• Structured tool / function calling in llama.cpp-style runtimes
• Agent-style multi-step interactions
• Small-footprint on-device or edge deployment

It is not intended to be a general replacement for larger Qwen models.

Sibling artifacts

• BF16 model + validation table: iromu/Qwen3-0.6B-tools
• NVFP4 checkpoint: iromu/Qwen3-0.6B-tools-NVFP4

Usage

Load a GGUF quant with llama.cpp:

llama-cli -hf iromu/Qwen3-0.6B-tools-GGUF:Q4_K_M

Or with the llama.cpp Python bindings:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_path="iromu/Qwen3-0.6B-tools-GGUF",
    filename="Qwen3-0.6B-tools-Q5_K_M.gguf",
    n_gpu_layers=99,
)

Validation

Measured on the sft_tools held-out validation split, greedy decoding. Throughput is single-stream greedy decode, not serving throughput.

suppressed in the Δ column means no comparison was admissible, which is a statement about the provenance of the two rows and not a measurement of no change. The reason is printed in the cell.

The UD-* rows are imatrix-guided (Dynamic) quants, distinct from the uniform quants: their per-tensor layout allocates bits by activation importance, and the importance matrix was calibrated on the SFT train split under the tag's pinned chat template — never on the validation split this table scores on.