Distil-Qwen3-1.7B-Customer-Support-Deferral (GGUF)

GGUF build of distil-labs/distil-qwen3-1.7b-customer-support-deferral, for serving with llama.cpp.

A fine-tuned Qwen3-1.7B model for multi-turn airline customer support that runs as the small tier of a two-model cascade: it handles most support turns itself and defers genuinely-hard turns to a larger model by emitting a defer_to_larger_model tool call. Every assistant action is a single tool call, including talking to the customer via respond_to_user, so a thin orchestrator can drive it.

Results

Evaluated on a held-out set of airline customer-support turns, scored by an independent GLM-5 judge (score = fraction of responses rated correct).

Fine-tuning lifts the local 1.7B from 0.42 to ~0.75 (closing roughly 85% of the gap to its frontier-scale teacher), while running ~96% of turns locally and escalating only the hardest ~4% to the larger model. The escalation is a cost/safety mechanism, not a quality boost over the small model alone.

Usage (llama.cpp)

hf download distil-labs/distil-qwen3-1.7b-customer-support-deferral-gguf \
    distil-qwen3-1.7b-customer-support-deferral.gguf --local-dir models

llama-server \
    --model models/distil-qwen3-1.7b-customer-support-deferral.gguf \
    --port 8000 \
    --jinja

Then query the OpenAI-compatible API at http://127.0.0.1:8000/v1. The airline policy (system prompt) and the 16 tool schemas ship with the demo app as job_description.json.

Demo App

This model powers the Flexible Customer Support Bot demo, a terminal cascade where this local SLM handles most airline-support turns and defers hard turns to a larger, OpenAI-compatible model.

File

Links

• Transformers model (safetensors)
• Demo app
• Distil Labs Website
• Hugging Face

License

Released under the Apache 2.0 license. See the transformers model card for base-model and teacher-model license terms.